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Abstract 


This  document  outlines  the  experimental  plan  for  the  first  Canadian  Virtual  Battle 
Experiment,  VBE  CA-1.  It  is  intended  to  provide  some  context  for  the  experiment  and 
act  as  a  blueprint  and  guide  for  its  implementation  and  execution. 

VBEs  are  being  used  by  the  Maritime  Systems  Group  Technical  Panel  1  of  The 
Technical  Cooperation  Panel  (TTCP  MAR  TP-1)  to  investigate  the  influence  of 
Network  Enabled  Capability  in  a  modular  synthetic  maritime  environment.  VBE 
CA-1  is  the  first  of  a  series  of  experiments  investigating  the  effect  of  sharing  low-level 
passive  sonar  data.  It  was  designed  to  use  multiple  subjects  in  multiple  sessions  of  a 
human-in-the-loop  experiment  to  produce  statistically  relevant  results. 


Resume 


Le  present  document  donne  un  apergu  du  plan  d’ experimentation  pour  la  premiere 
experience  de  combat  virtuel  du  Canada,  VBE  CA-1.  II  fournit  le  contexte  de 
T  experience  et  constitue  un  plan  et  un  guide  pour  sa  mise  en  oeuvre  et  son  execution. 

Les  VBE  sont  utilisees  par  le  Comite  technique  1  du  Groupe  d'analyse  des  systemes  de 
marine  du  Programme  de  cooperation  technique  (TP-1  MAR  TTCP)  pour  etudier 
l’influence  de  la  capacite  reseau  dans  un  environnement  maritime  synthetique 
modulaire.  La  VBE  CA-1  est  la  premiere  d’une  serie  d’experiences  sur  les 
consequences  du  partage  de  donnees  de  sonar  passif  a  faible  puissance.  Elle  a  recours 
a  des  sujets  multiples  dans  des  sessions  multiples  a  intervention  humaine  afin  de 
produire  des  resultats  pertinents  sur  le  plan  statistique. 


DRDC  Atlantic  TM  2003-158 


I 


This  page  intentionally  left  blank. 


DRDC  Atlantic  TM  2003-158 


Executive  summary 


Introduction 

This  document  outlines  the  experimental  plan  for  the  first  Canadian  Virtual  Battle 
Experiment,  VBE  CA-1.  It  was  intended  to  provide  some  context  for  the  experiment 
as  well  as  a  blueprint  and  guide  for  its  implementation  and  execution. 

VBEs  are  being  used  by  the  Maritime  Systems  Group  Technical  Panel  1  of  The 
Technical  Cooperation  Panel  (TTCP  MAR  TP-1)  to  investigate  the  influence  of 
Network  Enabled  Capability  in  a  modular  synthetic  maritime  environment.  VBE 
CA-1  is  the  first  of  a  series  of  experiments  investigating  the  effect  of  sharing  low-level 
passive  sonar  data.  It  was  designed  to  use  multiple  subjects  in  multiple  sessions  of  a 
human-in-the-loop  experiment  to  produce  statistically  relevant  results. 

Results 

This  document  provides  a  detailed  plan  for  the  implementation  of  VBE  CA-1.  It 
describes  the  objectives  and  requirements  for  the  experiment  and  the  infrastructure  to 
be  used  in  its  implementation.  Procedures  for  the  execution  of  the  experiment  and  the 
use  of  human  subjects  are  described.  Plans  for  the  collection  and  analysis  of  the 
experimental  data  and  the  dissemination  of  the  experimental  results  are  also  provided. 

This  document  provides  a  template  for  the  planning  and  implementation  of  follow-on 
experiments  in  this  or  similar  programs. 

Significance 

The  use  of  human  subjects  mandated  a  Human  Research  Ethics  Committee  (HREC) 
review  of  this  experiment.  This  was  the  first  HREC-reviewed  experiment  completed 
at  DRDC  Atlantic.  This  experiment  was  the  first  VBE  run  at  DRDC  Atlantic  in 
conjunction  with  a  current  international  series  of  TTCP  MAR  TP-1  experiments,  and 
the  first  to  make  use  of  the  new  Virtual  Combat  Systems  (VCS)  Group  laboratory. 

Future  plans 

This  experiment  is  the  first  in  a  series  aimed  at  investigating  how  the  value  of  shared 
coalition  data  changes  as  the  exchanged  data  becomes  increasingly  refined. 

Subsequent  experiments  will  build  on  the  results  of  this  experiment  and  investigate 
data  sharing  with  differing  sets  of  tools  and/or  information. 


Mellema,  G.R.  and  Wentzell,  T.E.  2004.  Experimental  Plan  for  VBE  CA-1. 
DRDC  Atlantic  TM  2003-158.  Defence  R&D  Canada  -  Atlantic. 


iii 


DRDC  Atlantic  TM  2003-158 


Sommaire 


Introduction 
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Resultats 

Le  present  document  offre  un  plan  detaille  de  la  mise  en  oeuvre  de  la  VBE  CA-1.  II 
decrit  les  exigences  et  les  objectifs  relatifs  a  1’ experience  ainsi  que  1’ infrastructure 
entourant  sa  mise  en  oeuvre.  Les  procedures  a  suivre  pour  1’ execution  de  1’ experience 
et  le  recours  a  des  sujets  humains  y  sont  decrites.  II  comprend  egalement  les  plans  de 
cueillette  et  d’analyse  de  donnees  d’ experimentation  et  de  diffusion  des  resultats  de 
T  experience. 

Le  document  fournit  un  modele  pour  la  planification  et  la  mise  en  oeuvre 
d’experiences  ulterieures  du  present  programme  ou  d’autres  programmes  similaires. 

Portee 

Le  recours  a  des  sujets  humains  a  demande  une  revision  de  1’ experience  par  le  Comite 
d'ethique  en  matiere  d'etude  sur  des  sujets  humains  (CEESH).  II  s’agissait  de  la 
premiere  experience  revisee  par  le  CEESH  a  etre  complete  a  RDDC  Atlantique,  et  de 
la  premiere  experience  VBE  menee  a  RDDC  Atlantique  dans  le  cadre  de  la  serie 
d’experiences  internationales  en  cours  du  TP-1  MAR  TTCP,  et  la  premiere  experience 
a  utiliser  le  nouveau  laboratoire  du  groupe  des  systemes  de  combat  virtuel  (SCV). 

Travaux  futurs 

L’ experience  est  la  premiere  d’une  serie  qui  vise  a  etudier  de  quelle  fagon  la  valeur  des 
donnees  de  coalition  change  lorsque  les  donnees  echangees  deviennent  de  plus  en  plus 
precises.  Les  experiences  qui  suivront  se  feront  a  partir  des  resultats  d’experiences  et 
serviront  a  etudier  le  partage  de  donnees  faisant  appel  a  differents  outils  ou  a 
differentes  informations. 
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1.  Introduction 


1.1  Coalition  Data  Exchange 

Information  exchange  among  coalition  partners  has  the  potential  to  provide  significant 
advantages  in  the  prosecution  of  underwater  warfare  [1].  The  degree  to  which  that 
potential  is  realized  however  may  be  dependent  on  the  available  bandwidth  between 
the  platforms  and  their  ability  to  process  the  received  data.  As  sensor  data  is  refined 
from  sound  pressure  to  target  location,  course  and  speed,  the  data  volume  decreases 
and  its  information  content  increases.  One  cost  is  latency.  Another  is  the  lost 
opportunity  to  undertake  lower-level  multi-sensor  processing,  such  as  cross-correlation 
or  triangulation. 

Exchanging  sensor-level  sonar  data  entails  its  own  set  of  costs  and  complexities.  High 
bandwidth  connections  as  well  as  powerful  and  sophisticated  processing  techniques 
are  essential  to  the  successful  utilization  of  multiple  streams  of  sensor-level  data.  In 
return  one  gains  the  capability  to  significantly  reduce  the  time  required  to  achieve 
target  localization  and  identification.  The  payoff  is  not  infinite  however;  at  some  level 
the  costs  of  increased  data  sharing  outweigh  the  additional  benefits,  due  primarily  to 
diminishing  returns.  In  order  to  make  a  good  decision  as  to  the  most  appropriate  level 
at  which  to  share  data  between  platforms,  one  needs  a  sense  of  how  these  costs  and 
benefits  trade  off. 

The  minimum  level  at  which  it  is  beneficial  to  share  data  depends  strongly  on  the 
speed  and  sophistication  of  the  data  processing  available  at  the  source  and  the  input 
requirements  of  the  processing  routines  at  the  recipient.  Although  the  minimum  input 
level  of  an  automated  processing  system  can  be  clearly  specified,  in  the  case  of  a 
human  operator  the  minimum  beneficial  input  level  may  be  more  difficult  to  identify, 
as  it  may  be  masked  by  issues  related  to  operator  loading  and  comprehension. 

An  investigation  of  the  potential  benefits  of  low-level  inter-platform  sonar  data 
exchange  requires  the  examination  of  a  series  of  scenarios  in  which  sonar  data  at 
differing  levels  of  development  are  exchanged.  The  value  of  the  exchanged  data  can 
be  assessed  in  terms  of  the  quality  of  development  of  the  local  operating  picture  [11]. 
The  experiment  described  in  this  document  is  the  first  experiment  in  a  series  aimed  at 
conducting  such  an  investigation. 


1 .2  Passive  Sonar  T rack  Association 

A  single  target  vessel  will  typically  produce  acoustic  emissions  at  multiple  frequencies 
[14]  [19].  These  signals  may  propagate  along  multiple  paths  to  a  receiver  and  may  be 
intermittent  in  time  due  to  such  external  influences  as  variations  in  source  or  receiver 
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location  or  the  local  environment.  Each  emission  is  eventually  represented  over  time 
to  the  sonar  operator  as  a  signal  track  segment1  from  a  potentially  unique  source. 

Passive  sonar  association  is  the  process  of  associating  signal  track  segments  across 
time,  frequency  or  bearing  into  master  tracks  that  are  common  to  a  single  source  vessel 
[9].  Although  rudimentary  tools  exist  to  assist  the  operator  in  this  task,  it  can  be  very 
labour  intensive  and  relies  heavily  on  the  operator’s  training  and  experience.  The  use 
of  an  automated  passive  sonar  association  system  or  aid  has  the  potential  to  reduce  this 
workload.  Improved  situational  awareness  in  the  underwater  environment  and  a 
reduction  in  the  response  time  to  threats  may  be  expected  by  increasing  the 
effectiveness  of  the  human  operator  and  the  potential  use  of  information  not  previously 
available  [10][17]. 

The  development  of  composite  master  tracks  from  multiple  track  segments  is  an 
inverse  problem.  From  knowledge  of  the  local  environment  and  the  source  and 
receiver  locations  it  can  be  fairly  straightforward  to  calculate  the  resultant  track 
segments,  but  extremely  difficult  to  reliably  determine  whether  apparent  relationships 
between  multiple  track  segments  correspond  to  a  common  origin. 

In  order  to  provide  some  insight  into  potential  processes  for  automated  track  segment 
association,  it  is  necessary  to  understand  the  decision  process  by  which  a  human 
operator  decides  whether  or  not  to  associate  track  segments  that  appear  to  have 
originated  from  the  same  source  into  master  tracks.  The  experiment  described  in  this 
document  is  the  first  in  a  series  of  experiments  aimed  at  investigating  that  process. 

1.3  TTCPVBEs 

The  Technical  Cooperation  Program  (TTCP)  Maritime  Systems  Group  (MAR) 
Technical  Panel  1  (TP-1)  was  stood  up  to  study  issues  of  relevance  to  Command  and 
Control  (C2)  and  Information  Management  (IM).  A  specific  goal  of  the  group  is  to 
study  the  effects  of  coalition  data  sharing  on  tactical  picture  development.  To  achieve 
this,  a  series  of  Virtual  Battle  Experiments  (VBEs)  are  being  be  performed  by 
participating  nations,  as  well  as  by  the  MAR  TP-1  group  as  a  whole.  Prior  to  the  start 
of  the  experiment  described  here,  Australia  had  completed  two  VBEs  addressing  high 
level  command  and  control  issues  and  was  planning  for  a  third  [6]  [7].  These 
experiments  demonstrated  that  VBEs  are  a  valid  method  for  addressing  command  and 
control  issues.  The  MAR  TP-1  group  also  collaboratively  planned  and  executed  two 
joint  international  VBEs,  VBE-B  and  VBE-C,  which  took  place  in  May  2003  and 
April  2004  respectively. 

The  infrastructure  developed  for  TTCP  VBEs  provides  a  synthetic  environment  and 
combat  system  testbed  in  which  the  sonar  track  sharing  issue  can  be  studied.  The 
implementation  of  this  experiment  also  provided  the  opportunity  for  Canada  to 
exercise  the  infrastructure  on  its  own  and  to  contribute  to  the  international  effort.  This 
experiment  was  the  first  of  a  series  of  experiments  aimed  at  fulfilling  that  role. 


1  A  signal  track  segment  is  a  single  sequence  of  bearing  and  frequency  values  in  time,  typically  plotted 
on  a  bearing-time  or  frequency-time  display,  representing  the  apparent  track  of  an  acoustic  source. 
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1.4  VBE  Personnel 


This  experiment  is  a  joint  effort  of  the  Virtual  Combat  Systems  (VCS)  and  the 
Underwater  Warfare  and  Data  Fusion  (UDF)  groups  at  Defence  R&D  Canada  - 
Atlantic.  The  VCS  group  at  Defence  R&D  Canada  is  involved  in  learning  about  the 
appropriate  application  of  the  synthetic  environments  to  problems  of  interest  to  the 
naval  community  and,  in  particular,  to  the  work  programs  of  other  scientists.  In 
September  2002,  the  UDF  group  was  approached  for  a  question  that  might  be 
answered  by  these  means.  It  is  hoped  that  in  the  future,  the  VCS  group  will  become  a 
known  resource  to  the  lab  (and  perhaps  beyond),  and  that  the  group  will  be  approached 
whenever  experimentation  is  seen  as  a  possible  means  of  answering  a  question  or 
testing  a  new  idea. 

The  main  personnel  involved  in  VBE  CA-1,  their  positions  and  their  responsibilities 
are  listed  in  Table  1. 


Table  1.  VBE  CA-1  Personnel. 


NAME 

POSITION 

RESPONSIBILITY 

Garfield  Mellema 

UDF  group  (defence  scientist) 

Lead  experimental  design,  run  director 

Tania  Wentzell 

VCS  group  (defence  scientist) 

Experimental  design,  run  director 

Jason  Murphy 

VCS  group  (computer  scientist) 

Software  development  and  implementation 

Okan  Topcu 

VCS  group  (NATO  exchange  officer) 

Verification  and  validation 

Dave  Hackett 

VCS  group  (Contractor) 

Software  development 

Mark  Hazen 

Leader  of  Virtual  Combat  Systems  (VCS)  group 

Bill  Roger 

Leader  of  Underwater  Warfare  Data  Fusion  (UDF)  group 
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2 .  Experiment O  bjectives and Conceptual  M  ode  I 


2.1  Focus,  Objectives  and  Conceptual  Model 

2.1.1  Experimental  Considerations 

Virtual  Battle  Experiment  CA-1  uses  simulated  maritime  scenarios  to  provide 
information  on  the  influence  of  shared  coalition  data  on  the  effectiveness  of  a  human 
sonar  operator.  The  successful  completion  of  this  experiment  requires  that  the  key 
aspects  be  identified  and  addressed  prior  to  its  execution  [13]  [18]. 

In  order  for  this  experiment  to  be  effective,  the  operator  must  be  presented  with  one  or 
more  maritime  scenarios  that  use  passive  sonar  to  address  a  plausible  operational 
requirement.  The  scenarios  also  need  to  be  realistic  enough  that  the  experimental 
issues  are  well  represented,  but  any  additional  details  or  procedures  beyond  that  would 
only  add  to  the  complexity  and  risk  of  the  experiment. 

In  order  for  the  experiment  to  address  the  issue  of  shared  sonar  data,  the  scenarios 
must  have  a  plausible  source  of  potentially  useful  nonorganic  sonar  data.  The 
scenarios  also  need  to  allow  for  the  insertion  of  the  nonorganic  data  without  other 
significant  changes  to  the  scenario  in  order  to  allow  an  unencumbered  comparison  of 
cases  with  and  without  the  shared  data. 

The  scenarios  must  include  an  operator  tasked  to  undertake  some  action,  the  outcome 
of  which  is  clearly  dependent  on  the  organic  and  nonorganic  data  and  the  operator’s 
comprehension  of  both.  There  should  also  be  some  metrics  with  which  to  evaluate  the 
differences  in  the  outcomes  of  the  two  types  of  scenarios. 

The  scenario  needs  to  provide  the  sonar  operator  with  sonar  track  segments,  a  motive 
for  associating  them,  a  tool  with  which  to  associate  them,  and  some  method  of 
querying  the  operator’s  rationale  when  they  are  associated  or  disassociated.  The 
degree  to  which  the  track  segments  are  associated  and  the  accuracy  of  the  associations 
are  useful  metrics  by  which  to  infer  the  level  of  operator  comprehension. 

The  experiment  should  encourage  the  collection  of  freeform  subject  feedback  in  order 
to  raise  subjective  issues  about  the  experiment  and  possibly  suggest  additional  ways  in 
which  the  outcome  of  the  experiment  might  be  evaluated.  The  collection  of  user 
feedback  also  recognizes  the  capabilities  of  the  operators,  and  their  suggestions  could 
include  ways  of  improving  future  experiments. 


2.1.2  Objectives 

The  objectives  of  this  experiment  are  to  address  the  following  questions: 
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1.  How  does  the  presence  of  nonorganic  track-level  bearings-only  sonar  information 
influence  sonar  operator  effectiveness  as  represented  by  manual  operational 
picture  development? 

2.  What  is  the  rationale  used  by  a  sonar  operator  to  decide  when  to  associate  or 
disassociate  passive  sonar  tracks? 

The  first  objective  presents  the  experiment  as  a  hypothesis  test  to  decide  whether  or 
not  the  presence  of  nonorganic  sonar  data  changes  the  outcome  of  the  experimental 
scenario.  The  answer  could  be  found  by  comparing  appropriately  chosen  metrics 
describing  the  state  of  development  of  the  operational  picture  with  and  without  an 
additional  display  presenting  nonorganic  data.  Further  consideration  of  these  and 
other  metrics  could  yield  a  more  complete  answer  as  to  the  direction  and  degree  of 
influence. 

The  sonar  time-bearing  plot  was  chosen  to  represent  the  operational  picture  in  this 
experiment  because  it  provides  a  sufficient  representation  of  the  operator’s  level  of 
understanding  of  the  situation.  The  development  of  bearings-only  sonar  tracks  into 
estimates  of  absolute  position  would  have  added  unnecessary  complexity  to  the 
operator’s  tasks. 

The  second  objective  is  a  discovery  exercise,  aimed  at  investigating  the  decision 
making  process  used  by  a  sonar  operator  in  the  course  of  assembling  an  operational 
picture.  The  track  segment  association  process  is  not  easily  described,  as  it  is 
influenced  to  varying  degrees  by  many  different  factors.  The  goal  here  is  to  collect 
information  about  the  specific  process  used  in  each  of  many  different  track 
associations.  This  information  can  then  be  structured  and  analyzed  to  better 
understand  the  decision  making  process. 


2.1.3  Conceptual  Model 

This  experiment  is  executed  as  multiple  independent  runs  of  a  computer  simulation, 
each  using  a  different  human  subject,  with  the  objective  of  providing  statistically 
relevant  results.  The  role  of  the  subject  in  this  experiment  is  that  of  a  sonar  operator 
onboard  a  frigate  using  a  towed  array  sonar  to  monitor  vessel  traffic  in  a  narrow  strait. 
The  operator’s  task  is  to  develop  as  complete  an  operational  picture  as  possible  using 
only  passive  broadband  sonar  information  from  the  ownship  and  allied  positions. 

Passive  sonar  data  is  often  presented  to  an  operator  in  bearing  and/or  frequency  versus 
time  format  as  a  time-sequence  of  independent  scans.  The  presence  of  a  nearby  vessel 
is  typically  indicated  by  repeated  intensity  peaks  at  bearings  and  frequencies 
corresponding  to  the  propagation  paths  and  characteristic  emissions  from  that  vessel. 
After  identifying  repetitive  intensity  peaks  in  the  scans  as  track  segments,  the  operator 
can  then  proceed  to  associate  those  track  segments  that  are  believed  to  have  a  common 
origin  across  time,  frequency  and  bearing.  These  time-consuming  tasks  require  some 
level  of  training  and  skill  on  the  part  of  the  operator  but  the  manual  performance  of 
most  of  these  tasks  was  not  relevant  to  this  experiment.  In  order  to  limit  the 
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complexity  of  this  experiment  without  significantly  diminishing  its  relevance,  the 
passive  sonar  data  is  presented  to  the  operator  already  in  track  segment  form. 

The  complexity  of  the  passive  sonar  picture  can  be  altered  by  changing  the  number  of 
targets,  the  number  of  propagation  paths  between  the  target  vessels  and  the  receiver,  or 
the  number  of  tonals2  produced  by  each  target  in  a  narrowband  scenario. 
Comprehension  of  a  picture  complicated  by  the  multipath  propagation  is  likely  to  be 
beyond  the  ability  of  a  novice,  untrained  subject.  The  use  of  narrowband  scenario 
presenting  the  operator  with  frequency  as  well  as  bearing  information  would  require 
that  the  operator  be  presented  with  a  much  more  complex  display  than  in  the 
broadband  case.  The  use  of  multiple  targets,  each  responsible  for  no  more  than  one 
displayed  track  segment  at  a  time,  was  deemed  to  make  the  scenarios  sufficiently 
challenging  to  the  operator  without  adding  undue  complexity  to  the  overall 
experiment.  The  sonar  operator  is  therefore  required  only  to  associate  the  track 
segments  across  time,  not  bearing  and  no  frequency  information  is  presented. 
Characterization  of  the  multiple  targets  as  anything  other  than  surface  vessels  was  also 
deemed  unnecessary. 

In  order  to  adequately  evaluate  the  influence  of  sonar  track  sharing,  the  shared  sonar 
track  segments  must  contain  data  that  is  potentially  valuable  to  the  operator  and  be 
available  in  a  familiar  format.  In  order  to  provide  nonorganic  data  necessary  to  this 
experiment,  a  geographically  remote  allied  vessel  using  a  similar  sonar,  also  providing 
full  coverage  of  the  travelled  portion  of  the  strait  but  from  a  different  perspective,  was 
included  in  the  scenario.  A  navigational  chart  display,  which  indicated  the  position  of 
the  ownship  in  the  strait,  also  showed  the  position  of  the  allied  ship.  No  other  vessel 
information  is  available  from  the  navigational  chart  display.  In  those  cases  where  the 
operator  is  provided  with  nonorganic  data,  a  third  display  shows  the  same  type  of 
sonar  track  segments,  but  from  the  perspective  of  the  allied  ship.  The  difference  in  the 
two  perspectives  is  the  bearing  to  targets  and  the  detection  ranges. 

In  a  typical  operational  scenario,  a  vessel  towing  a  line  array  needs  to  change  course 
periodically  in  order  to  resolve  bearing  ambiguities  inherent  to  the  use  of  a  towed  array 
sonar.  In  this  experiment  both  the  ownship  and  the  allied  ship  make  occasional  course 
changes  of  appropriate  magnitude  at  appropriate  intervals,  but  the  sonar  operator  has 
no  ability  to  influence  these  manoeuvres.  Although  the  ability  to  influence  ownship 
manoeuvres  could  improve  the  operator’s  ability  to  discriminate  targets,  providing  this 
ability  would  have  undermined  the  controlled  conditions  and  repeatability  of  the 
experiment.  In  addition  the  courses  of  the  target  vessels  are  significantly  constrained 
by  the  limited  navigable  waters  of  the  strait. 

The  development  of  metrics  by  which  to  evaluate  the  operational  picture  developed  by 
the  sonar  operator  requires  that  truth  data  be  available  for  processing  following  the 
experiment.  This  data  was  identified  as  the  correspondence  between  the  target  vessels 
and  the  primary  track  segments  presented  to  the  operator,  and  the  track  components  of 
each  track  association  and  disassociation  decision.  The  timings  of  each  track  segment 
initiation  and  each  association  or  disassociation  decision  are  also  included. 


2  single-frequency  tones 
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The  operator  decision  rationale  objective  requires  that  the  operator  be  queried 
following  each  association  or  disassociation  decision  as  to  the  reason  for  that  decision. 
The  query  needs  to  be  immediate  but  brief  and  unobtrusive  enough  that  the  operator’s 
concentration  was  not  significantly  disrupted. 
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3.  Experimental  Requirements 


This  experiment  investigates  the  interaction  between  a  sonar  operator  and  a  passive 
sonar  bearing-time  display  presented  in  a  synthetic  maritime  environment.  The 
conceptual  model  in  the  previous  chapter  described  the  overall  interaction  of  the  key 
parts  of  the  experiment:  the  simulation  infrastructure,  the  scenarios  and  the  subject 
operators.  In  this  chapter,  the  core  requirements  of  each  of  those  parts  will  be 
specified  explicitly. 


3.1  Simulation  Infrastructure  Requirements 

The  simulation  infrastructure  is  used  to  present  the  sonar  operator  with  a  realistic 
representation  of  maritime  tactical  data  and  to  record  the  interaction  of  the  operator 
with  that  environment.  The  infrastructure  must  also  support  the  experimental 
scenarios.  The  requirements  of  the  infrastructure,  based  on  the  conceptual  model,  are 
as  follows. 

1.  Show,  on  a  navigational  chart  display,  the  position  and  movement  of  at  least 
10  vessels  following  paths  at  predetermined  courses  and  speeds.  Be  able  to 
restrict  the  information  shown  to  that  of  the  ownship  and  allied  ship.  The 
positions  of  all  of  the  vessels  will  be  logged. 

2.  Show,  on  a  passive  sonar  bearing-time  display,  from  the  perspective  of  a 
designated  ownship,  broadband  signal  tracks  corresponding  to  the  other 
nearby  vessels. 

3.  Show,  on  a  second  passive  sonar  bearing-time  display,  from  the  perspective  of 
a  designated  allied  ship,  broadband  signal  tracks  corresponding  to  the  other 
nearby  vessels. 

4.  The  broadband  sonar  tracks  will  be  plotted  as  time  versus  bearing.  The  direct 
path  bearings  will  be  calculated  according  to  straight-line  paths.  The  signals 
received  from  each  nearby  source  will  be  updated  periodically  and  the 
corresponding  tracks  will  terminate  and  reinitiate  according  to  the  following 
formula. 

a.  A  track  will  be  initiated  when  the  signal-to-noise  ratio  (SNR)  of  the 
calculated  signal  exceeds  a  specified  threshold  m  times  within  n  updates. 
The  signal  source  level  will  be  determined  by  the  source  vessel  type  and 
attenuation  due  to  spreading  along  the  propagation  path  between  the 
source  and  receiver.  The  attenuation  will  be  calculated  as  15  log  R ,  where 
R  is  the  length  of  the  straight-line  path  between  the  source  and  receiver.  A 
combination  of  fixed  and  random  amounts  of  background  noise  will  also 
be  added  to  the  signal. 
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b.  The  track  will  be  terminated  when  the  SNR  of  the  received  signal  falls 
below  the  detection  threshold  for  a  specified  number  of  consecutive 
updates.  The  track  segments  will  be  assigned  unique  identifiers.  The 
signal  and  noise  levels  will  be  logged. 

5.  The  track  displays  will  include  tools  to  permit  an  operator  to  identify 
individual  track  segments  and  associate  them  into  a  fused  master  track.  The 
operator  will  also  be  able  to  disassociate  a  master  track  back  into  its 
component  tracks.  The  association  or  disassociation  time  and  the  track 
identifiers  will  be  logged. 

6.  The  operator  will  be  queried  following  each  association  or  disassociation 
decision  as  to  the  rationale  behind  that  decision  and  the  results  of  that  query 
will  be  logged. 

7.  Since  this  experiment  will  investigate  influence  of  a  very  rudimentary  level  of 
data  sharing,  it  will  not  be  possible  for  the  operator  to  electronically  associate 
tracks  between  the  ownship  and  allied  ship  track  displays.  The  operator  must 
make  any  associations  between  the  data  on  the  two  displays  cognitively  and 
apply  these  association  to  either  or  both  displays. 

8.  The  operator  will  be  able  to  adjust  the  origin  and  scale  of  the  navigation  and 
track  displays. 

9.  The  simulation  and  the  operator  interface  must  run  in  real  time. 

10.  The  stimuli  presented  to  the  operator  and  the  response  of  the  console  to 
operator  input  must  be  predictable  and  repeatable. 

11.  The  passive  sonar  chart  displays  must  be  synchronized  to  each  other  and  the 
navigational  chart  display  at  all  times. 

12.  The  navigational  and  chart  displays  must  appear  on  physically  separate 
windows  so  that  they  can  be  viewed  simultaneously. 

13.  All  log  files  must  use  a  common  clock. 

14.  The  Run  Director,  who  is  responsible  for  the  execution  of  the  experimentation 
session,  must  be  able  to  monitor  the  operation  and  performance  of  the 
simulation,  particularly  the  simulation  time  and  the  simulation  time  rate. 

15.  The  number  of  vessels  and  their  positions,  courses,  speeds  and  source  levels 
must  be  specifiable  by  the  Run  Director  prior  to  an  experimentation  run.  The 
background  noise  levels  will  also  be  specifiable. 

16.  The  simulation  infrastructure  must  support  at  least  2,  but  preferably  4, 
independent  operators  as  well  as  the  Run  Director.  The  log  files  of  each  user 
must  be  distinct. 
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3.2  Scenario  Requirements 


A  minimum  of  three  maritime  scenarios  will  be  required  for  this  experiment,  one  of 
which  will  be  used  for  subject  training  and  demonstrations.  Since  each  subject  will  be 
participating  in  two  experimentation  sessions,  one  with  shared  coalition  data  and  the 
other  without  shared  data,  it  is  important  that  different  scenarios  be  presented  in  each 
session  in  order  to  reduce  the  influence  of  learning  on  the  results  of  the  second  session. 
Additional  scenarios  may  also  be  useful  as  spares,  but  their  use  should  be  sufficiently 
distributed  that  later  analysis  would  be  able  to  indicate  whether  the  sequence  in  which 
the  scenarios  were  presented  was  a  relevant  factor  in  the  experimental  results.  In  any 
case,  the  scenarios  must  be  easily  comprehended  by,  and  present  clear  objectives  to, 
operators  with  varying  levels  of  expertise. 

The  scenarios  used  in  this  experiment  must  include  an  ownship  and  an  allied  ship 
manoeuvring  cooperatively  in  a  narrow  strait.  A  sufficient  number  of  other  vessels 
must  also  be  travelling  in  the  strait.  The  positions  and  movements  of  the  ownship  and 
allied  ship  must  be  conducive  to  the  monitoring  of  the  other  vessels  in  the  strait, 
preferably  providing  redundant  views  of  a  similar  subset  of  the  vessel  traffic.  It  is 
essential  that  the  sonar  tracks  produced  by  the  ownship  and  allied  ship  sonars  are 
sufficiently  intermittent  in  time  that  it  would  be  somewhat  difficult  for  the  operator  to 
identify  and  reassemble  segments  that  are  likely  to  have  originated  from  the  same 
vessel.  Ideally,  this  level  of  difficulty  of  this  task  would  vary  from  simple  and  obvious 
to  challenging  and  ambiguous. 

There  should  be  a  sufficient  amount  of  traffic  within  sonar  range  of  the  allied  vessels 
to  provide  a  complex  but  not  overwhelming  sonar  picture  to  the  operator.  The 
complexity  of  the  sonar  picture  may  be  due  in  some  part  to  the  presence  of  vessels 
having  similar  bearing  rates  at  differing  ranges  and  speeds.  The  allied  vessels  should 
make  periodic  turns  consistent  with  target  motion  analysis  (TMA)  by  a  towed  array 
equipped  vessel.  The  vessel  traffic  in  the  strait  should  be  at  least  bi-directional,  if  not 
multidirectional,  but  their  movement  must  be  consistent  with  that  expected  in  a 
realistic  maritime  scenario. 

Since  one  of  the  objectives  of  this  experiment  is  to  investigate  the  process  by  which  an 
operator  makes  association  decisions,  the  problem  of  correctly  associating  the  track 
segments  need  not  be  necessarily  tractable.  It  is  necessary  though,  to  know  the  origins 
of  the  track  segments  in  order  to  evaluate  the  effectiveness  of  the  various  decision¬ 
making  processes. 

3.3  Subject  Requirements 

The  subjects  to  be  used  as  operators  in  this  experiment  should  have  at  least  a 
rudimentary  understanding  of  how  sonar  is  used  to  track  sea  vessels.  They  should  also 
be  able  to  follow  the  progress  of  a  vessel  on  the  navigational  chart  display.  Neither 
training  nor  experience  in  tactical  sonar  operations  is  necessary  for  this  experiment, 
although  it  may  be  a  prerequisite  for  later  experiments  in  this  program. 


DRDC  Atlantic  TM  2003-158 


11 


The  ideal  candidate  for  this  experiment  would  be  technically  competent  in-house 
personnel  who  are  at  least  familiar  with  acoustics  and/or  have  naval  military  training. 

3.4  Experimentation  Plan  Milestones 

Table  2  provides  a  time  line  for  the  different  phases  of  this  experiment. 


Table  2.  VBE  CA-1  time  line 


TASK 

DATE(S) 

Orientation  Meeting 

November  2002 

Capabilities  Analysis 

December  2002 

Requirements  Analysis 

January  2003 

Initial  Infrastructure  Plan 

February  2003 

Conceptual  Model 

March  2003 

Draft  Experimental  Plan 

April  2003 

Submit  Proposal  to  the  Human  Research  Ethics  Committee  (HREC) 

May  2003 

Demonstrate  Simulation  Infrastructure 

June  2003 

HREC  Approval 

July  2003 

Demonstrate  Scenarios 

July  2003 

VBE  CA-1  Test  Run 

August  2003 

Call  for  Volunteers 

August  2003 

Final  Software  Modification 

September  2003 

Experimental  Plan  -  Final 

September  2003 

VBE  CA-1  with  Operators 

September  2003 

Completed  Analysis 

October  2003 

Scientific  Report  of  the  Results 

December  2003 

Completion  of  Documentation 

July  2004 
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4.  Simulation  Infrastructure 


The  experimentation  sessions  will  take  place  in  a  simulated  operations  room 
environment.  The  simulation  will  include  two  independent  sonar  operator’s 
workstations,  each  represented  by  a  pair  of  computer  displays,  one  showing  a 
navigational  chart  indicating  the  position  of  the  operator’s  own  ship  and  the  other 
showing  passive  sonar  tracks  from  the  ownship.  A  third  display,  showing  passive 
sonar  tracks  from  an  identically  equipped  but  geographically  remote  allied  vessel  may 
also  be  available  at  one  of  the  stations.  A  keyboard  and  mouse  will  be  available  for 
the  operator  to  interact  with  either  of  the  ownship  displays.  A  second  keyboard  and 
mouse  will  be  available  for  the  operator  to  interact  with  the  allied  ship  display.  The 
station  with  only  two  displays  is  designated  as  the  Solo  Subject  Station  while  the 
station  with  access  to  the  allied  sonar  display  is  designated  as  the  Coalition  Subject 
Station. 

The  information  provided  on  the  subject  station  displays  will  be  provided  by  maritime 
simulation  software  running  at  a  third  station,  designated  as  the  Run  Director  Station. 
From  this  station  the  Run  Director  will  initiate,  control  and  monitor  the 
experimentation  sessions.  The  simulation  software,  described  in  the  following  section, 
executes  predetermined  simulated  maritime  scenarios,  the  development  of  which  is 
described  in  the  following  chapter.  Each  of  the  simulation  interfaces  seen  by  the 
subjects  are  independent,  receiving  data  from  the  simulation,  but  not  providing  data  to 
the  simulation  or  each  other.  They  are  described  in  Section  4.2. 


4.1  Simulation  Tool  Components 

The  maritime  simulation  was  constructed  using  Virtual  Maritime  Systems 
Architecture3  (VMSA)  based  federates,  each  federate  providing  a  component  or 
subsystem  capability  to  the  simulation,  allowing  the  simulation  to  be  tailored  to  the 
needs  of  the  experiment  [3].  VMSA  is  High  Level  Architecture  (HLA)  compliant 
[21]. 

The  experiment  software  can  be  organized  into  two  parts,  the  first  part  dealing  with  the 
simulation  of  relevant  aspects  of  the  movement  and  acoustics  of  the  vessels,  producing 
both  vessel  movement  and  sonar  tracks,  and  the  second  part  displaying  those  tracks  for 
the  subjects  to  observe  and  interact  with.  For  best  performance,  all  components  of  the 
first  part  reside  on  the  Run  Director  Station  while  the  subject  stations  include  only 
instances  of  the  components  that  are  necessary  for  the  subject  user,  as  shown  in  Figure 
1.  This  also  permits  the  number  and  configuration  of  the  subject  stations  to  be 
independent  of  the  rest  of  the  simulation. 


3  VMSA  was  developed  by  the  Australian  Defence  Science  and  Technology  Organization  (DSTO). 
The  architecture  and  many  simulation  components,  called  federates,  were  made  available  to  DRDC 
Atlantic  through  a  Memorandum  of  Understanding  between  Canada  and  Australia,  Subsidiary 
Arrangement  18. 
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Figure  1.  The  simulation  infrastructure  can  be  separated  into  three  parts. 


The  base  component  of  the  software  running  on  the  Run  Director  Station  is  the  Run¬ 
Time  Interface  (RTI).  The  RTI  provides  an  interface  between  each  of  the  federates 
and  the  rest  of  the  federation.  The  Virtual  Maritime  System  Execution  Manager 
(VMSEM)  federate  manages  the  pace  at  which  the  federation  operates,  ensuring  that 
the  simulation  does  not  start  until  all  of  the  federates  have  joined  and  that  the 
federation  time  does  not  advance  more  rapidly  than  specified. 

VMSA-based  simulations  are  time  managed,  in  that  federates  can  be  time  constrained 
and  time  restricting,  allowing  them  to  be  well  controlled  and  repeatable.  By  taking 
advantage  of  this  feature,  the  results  of  multiple  independent  runs  of  an  experiment 
using  different  human  operators  can  be  analyzed  to  compile  statistically  relevant 
answers  to  address  an  experimental  objective.  Time  regulation  also  permits  Monte 
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Carlo  analysis  of  the  influence  of  changes  in  the  experimental  scenario,  such  as  the 
replacement  of  processing  algorithms  or  the  addition  of  operator  tools  or  shared 
coalition  data. 

In  the  VMSA,  the  course  and  speed  of  a  vessel  are  controlled  by  commands  from  a 
helm  federate  that  are  translated  into  position,  course  and  speed  updates  by  a  motion 
federate.  In  this  experiment  all  of  the  helm  commands  are  issued  by  an  automated 
helm  federate  called  Gameboard,  which  uses  an  XML  script  to  manoeuvre  vessels 
along  predefined  courses.  The  XML  scripts  are  generated  using  a  graphical  scripting 
tool  called  Scenario  Generator  [16],  as  described  in  a  later  chapter.  Although  a 
federate  exists  to  allow  helm  commands  to  be  issued  manually  by  an  operator,  this 
federate  is  not  used  since  the  scenarios  used  fixed  the  course  and  speed  of  each  ship. 
Allowing  the  operators  to  manoeuvre  the  ships  would  jeopardize  the  repeatability  of 
the  experimental  scenarios. 

The  original  sonar  federate  available  with  VMSA  is  unsuitable  for  this  experiment  as  it 
had  been  designed  for  use  in  scenarios  that  were  more  command  and  control  oriented, 
and  lacks  most  of  those  aspects  of  sonar  signals  that  make  them  challenging  to 
interpret  and  difficult  to  use.  It  was  therefore  necessary  to  construct  a  more  suitable 
sonar  federate  following  the  requirements  of  Section  3.1. 

The  role  of  the  sonar  federate  in  this  experiment  is  to  observe  the  locations  and  types 
of  each  of  the  vessels  and  use  that  information  to  produce  sonar  track  segments  for 
display  to  the  sonar  operator.  There  is  no  requirement  for  the  sonar  simulation  to 
follow  the  same  processing  steps  as  a  real  sonar  or  to  precisely  model  every  detail  of 
the  underwater  environment  so  long  as  the  end  result  has  characteristics  representative 
of  real  sonar  tracks  that  would  be  experienced  by  an  operator  during  a  sea  trial.  A 
basic  description  of  the  requirements  for  the  sonar  federate  in  this  experiment  is  as 
follows. 


The  sonar  federate  will: 

1.  calculate  the  signal  excess  of  the  received  broadband  acoustic 
emissions  from  each  target  vessel,  taking  into  account: 

•  the  typical  source  level  of  that  vessel  type, 

•  transmission  loss  calculated  using  a  spherical-cylindrical  model 
appropriate  for  medium  range  shallow  water  propagation, 

•  interfering  sources  such  as  other  nearby  vessels,  and 

•  ambient  noise,  including  randomly  distributed  fluctuations  in  the 
ambient  noise. 

2.  When  the  signal  excess  exceeds  a  specified  threshold  at  least  m  times 
within  n  simulation  time  steps,  a  sonar  track  segment  will  be  initiated 
at  a  bearing  corresponding  to  a  direct  path  arrival  from  that  vessel. 
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3.  Once  initiated,  the  track  will  be  regularly  updated  so  long  as  the 
signal  excess  does  not  fall  below  the  specified  threshold  for  more 
than  a  specified  number  of  consecutive  time  steps. 

4.  Although  there  is  no  limit  to  the  number  of  temporally  distinct  sonar 
track  segments  produced  by  each  target  vessel,  each  target  vessel  may 
produce  no  more  than  one  track  segment  at  any  time  and  all  track 
information  will  be  discarded  between  segments. 

5.  For  analysis  purposes,  the  sonar  federate  will  produce  a  log  file 
detailing  the  initiation  and  termination  times  and  conditions  of  each 
track,  the  track  update  values  and  the  identities  of  the  vessel  or 
vessels  to  which  each  track  corresponds. 

The  subject  stations  use  multiple  instances  of  Horizon,  a  track  data  hosting  and 
management  system  provided  by  DSTO  as  an  operator  interface  into  the  simulation 
[2].  Each  instance  of  Horizon  interfaces  directly  with  the  rest  of  the  federation  and 
each  has  a  separate  display.  No  information  is  passed  between  the  Horizon  instances 
or  from  Horizon  into  the  rest  of  the  federation  although  each  instance  of  Horizon 
produces  its  own  text  and  binary  log  files.  Both  the  solo  and  coalition  configurations 
of  the  operator  station  have  a  navigational  chart  display  and  a  sonar  display  showing 
passive  sonar  tracks  received  by  the  ownship.  The  coalition  configuration  includes  an 
additional  display  showing  passive  sonar  tracks  received  by  an  allied  ship. 

The  operator  can  associate  and  disassociate  sonar  tracks  in  Horizon,  but  Horizon  does 
not  query  the  operator  as  to  the  rationale  for  that  decision,  it  merely  acts  on  it.  An 
overlay  application,  called  EnterReason,  was  therefore  constructed  in-house  to  detect 
when  the  operator  has  clicked  in  the  screen  location  of  the  associate/disassociate 
button  and  present  a  multiple-choice  query  as  to  the  rationale  for  that  decision. 

The  Run  Director  Station  is  hosted  on  a  single  computer,  as  is  the  Solo  Subject 
Station.  The  Coalition  Subject  Station  is  hosted  on  a  pair  of  computers,  primarily  due 
to  the  requirement  for  three  instances  of  Horizon  and  three  displays.  Each  computer  is 
a  dual  processor  2.0  GHz  Xeon  with  1  GB  of  RAM  running  Windows  2000  and  the 
stations  are  networked  through  a  private  100  Mb/s  switch  to  prevent  interference  from 
outside  network  traffic.  This  ensures  that  the  experience  of  the  operators  will  be 
predictable  and  repeatable.  The  federation  rate,  which  is  defined  as  the  number  of 
simulation  time  steps  executed  by  the  federation  per  second,  is  measured  by  the 
execution  manger  and  shown  on  the  Run  Director  display  along  with  the  simulation 
time  step  count.  Real-time  performance  is  considered  to  be  one  time  step  per  second. 


4.2  Simulation  User  Interface 

This  experiment  compares  the  effectiveness  of  sonar  operators  having  access  to 
ownship  data  only  with  those  having  access  to  additional  data  from  an  allied  ship.  The 
experimentation  sessions  are  therefore  configured  so  that  the  only  difference  between 
the  two  types  of  operator  stations  is  the  presence  of  an  additional  sonar  track  display 
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Figure  2.  The  Horizon  chart  display  shows  the  local  navigable  waters  and  the  locations  of  the 
ownship  and  the  allied  ship. 


and  its  corresponding  keyboard  and  mouse  in  the  coalition  configuration  relative  to  the 
solo  configuration. 

Each  subject  station  has  a  navigational  chart  display  showing  the  local  navigable 
waters  and  the  positions  of  the  ownship  and  allied  ship  as  shown  in  Figure  2.  The 
operator  can  adjust  the  scale  and  origin  of  the  navigational  chart  display.  Although 
additional  vessel  traffic  is  present  in  the  local  area,  it  is  not  shown  on  the  navigational 
chart  display.  The  only  source  of  information  regarding  the  additional  vessel  traffic  is 
the  intermittent  sonar  track  segments  on  the  passive  sonar  display,  motivating  the  track 
association  process. 

Each  subject  station  has  an  ownship  sonar  display  showing  sonar  track  segments  that 
originated  from  the  ownship  towed  array  receiver.  An  example  of  a  sonar  chart 
display  is  shown  in  Figure  3.  The  ownship  heading  is  shown  as  a  series  of  filled  grey 
circles  while  each  sonar  track  segment  is  shown  as  a  series  of  open  yellow  circles.  A 
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Figure  3.  The  Horizon  time-bearing  display  shows  sonar  track  segments  such  as  these  produced  by 
the  ownship  sonar. 


question  mark  indicates  the  end  of  each  track  segment.  The  tracks  are  displayed  in 
bearing  versus  time  format  and  the  time  scale  of  the  display  is  user-adjustable  to  show 
30,  60  or  90  minutes  of  data.  The  user  can  also  zoom  in  on  a  particular  bearing  range 
by  dragging  across  it  with  the  mouse. 

Individual  sonar  track  segments  can  be  selected  and  deselected  by  clicking  on  them 
with  the  mouse  and  then  associated  into  a  fused  track  using  the  Active  Fusion  tool 
shown  in  the  lower  left  of  the  display.  Fused  track  segments  are  removed  from  the 
display  and  replaced  with  the  fused  track,  drawn  as  a  series  of  lightning  bolt  symbols. 
The  Active  Fusion  tool  can  also  be  used  to  add  additional  track  segments  into  a  fused 
track  or  to  disassociate  a  fused  track  back  into  its  component  track  segments. 

Although  the  operator  at  the  Coalition  Subject  Station  can  see  both  the  ownship  and 
allied  ship  sonar  displays  and  use  information  from  one  display  to  make  decisions 
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Figure  4.  The  EnterReason  popup  appears  whenever  an  association  or  disassociation  is  made. 


regarding  the  other,  information  cannot  be  electronically  associated  between  the 
displays. 

Both  the  association  and  disassociation  processes  require  that  the  operator  click  a 
button  in  a  unique  region  of  the  display.  Mouse  clicks  in  this  region  trigger  a  query 
tool  called  EnterReason,  which  queries  the  operator  as  to  the  rationale  that  led  to  the 
association  or  disassociation  decision.  Several  options  are  presented  in  a  multiple- 
choice  format,  as  shown  in  Figure  4,  with  a  final  choice  of  ‘other.’  Selecting  the 
‘other’  option  opens  a  text  box  in  which  to  elaborate.  The  options  are  read  from  a 
configuration  file  and  the  operator  responses  and  decision  times  are  logged  to  a  file. 
The  decision  times  are  used  to  specifically  identify  the  tracks  being  associated  or 
disassociated,  as  the  Horizon  and  EnterReason  programs  are  not  directly  interfaced. 
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5.  Experiment  Scenarios 


The  simulation  infrastructure  described  in  the  previous  chapter  provides  a  platform  in 
which  to  execute  maritime  scenarios.  It  is  the  position  and  movement  of  vessels  in 
these  scenarios  that  produces  the  simulated  maritime  environment  necessary  for  this 
experiment.  Five  scenarios  are  available,  one  for  training  and  demonstration  purposes, 
two  for  use  in  experimentation  sessions,  and  two  others  are  kept  in  reserve  as  spares  in 
case  of  unforeseen  problems  with  the  training  or  experimentation  scenarios.  Each  of 
the  maritime  scenarios  fulfills  the  requirements  specified  in  Section  3.2. 

The  primary  tool  for  scenario  development  is  Scenario  Generator,  which  produces  an 
XML  script  file  that  can  be  read  by  the  Gameboard  federate  to  control  the  movements 
of  the  ownship,  the  allied  ship  and  the  other  vessel  traffic.  Figure  5  shows  the  training 
scenario,  Tl,  in  Scenario  Generator.  Initial  analysis  of  the  scenarios  is  done  using  a 
tool  called  BearingGenerator,  which  reads  the  XML  script  files  and  translates  the 
vessel  positions  over  time  into  bearings  relative  to  any  other  vessel  over  time.  Figure 
6  shows  the  same  scenario  in  BearingGenerator.  The  bearing  tracks  produced  by 
BearingGenerator  are  continuous  in  time,  regardless  of  the  received  signal  level  or  the 
presence  of  interfering  sources. 

A  key  requirement  of  the  scenarios  is  the  development  of  temporally  distinct  sonar 
track  segments.  The  likelihood  of  a  track  being  temporally  disrupted  is  dependent  on 
several  factors  including  the  target  vessel  source  level,  the  nearby  presence  and 
strength  of  other  sources,  the  ambient  noise  level,  and  the  degrees  of  randomness  in 
the  background  noise.  All  of  these  factors  except  the  location  of  nearby  sources  can  be 
modified  in  the  configuration  file  of  the  sonar  federate.  The  source  level  of  a 
particular  vessel  can  be  modified  in  Scenario  Generator  by  specifying  the  class  of  that 
vessel,  e.g.  freighter  or  frigate,  or  in  the  sonar  configuration  file  by  specifying  the 
source  characteristics  of  that  class. 

The  experimental  scenarios  are  intended  to  be  difficult  enough  to  be  interesting,  but 
not  so  difficult  as  to  overwhelm  novice  operators.  In  the  development  of  a  typical 
scenario,  the  first  few  vessels  produce  sonar  tracks  that  are  relatively  distinct  and  can 
be  readily  identified  by  a  novice  operator.  As  additional  vessels  are  added,  some  of 
the  classic  complications  can  be  seen,  such  as  crossing  sonar  tracks,  near-coincident 
sonar  tracks  due  to  multiple  vessels  at  similar  bearings  with  similar  bearing  rates,  and 
the  potential  for  track  seduction,  where  a  sonar  track  segment  from  one  vessel  is 
terminated  while  a  sonar  track  segment  from  another  is  initiated. 

The  scenarios  are  developed  by  entering  the  configuration  and  routing  of  each  vessel 
in  Scenario  Generator,  generating  an  XML  script  file  and  then  observing  the  resultant 
bearing  tracks  in  BearingGenerator.  In  order  to  see  how  the  scenario  plays  out  with 
segmented  sonar  tracks,  it  is  necessary  to  enter  the  script  into  Gameboard,  run  the 
simulation  infrastructure  and  observe  the  results  in  Horizon.  Each  scenario  is  about  2 
hours  long  and,  due  to  limitations  of  the  VMSA,  the  simulation  can  only  be  run  from 
the  beginning  forward.  Although  the  track  generation  federates  can  be  run  at  higher 
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Figure  5.  The  scenarios  used  in  this  experiment  were  laid  out  using  Scenario  Generator.  The  training 
scenario,  T1,  is  shown  here. 


speed,  Horizon  cannot,  so  the  simulation  is  typically  run  without  continuous 
observation  while  the  scenarios  are  in  development.  Camtasia,  a  third-party  display 
recording  program  from  TechSmith,  was  be  used  to  record  the  displays  and  then  replay 
arbitrary  portions  of  the  scenario  at  variable  speed.  Following  each  simulation  run,  the 
scenarios  are  adjusted,  either  by  changing  the  vessel  characteristics  or  the  sonar 
configuration  until  the  scenario  is  deemed  to  be  satisfactory. 
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Figure  6.  Initial  analysis  of  the  scenarios  is  done  using  BearingGenerator,  which  reads  the  XML  script  files 
and  translates  the  vessel  positions  over  time  into  bearings  relative  to  any  other  vessel  over  time. 


The  iterative  nature  of  scenario  development  is  largely  due  to  the  use  of  random  values 
in  the  sonar  federate  and  the  lack  of  tools  to  invert  sonar  tracks  back  into  vessel  tracks. 
However,  the  techniques  used  are  sufficient  for  the  purposes  of  this  experiment.  Since 
one  of  the  objectives  of  this  experiment  is  to  investigate  the  process  by  which  an 
operator  makes  association  decisions,  there  is  no  requirement  for  a  single,  unique 
solution  to  the  association  problem.  Each  of  the  scenarios  uses  about  10  vessels 
including  the  ownship  and  the  allied  ship,  and  each  of  the  two  sonars  produces 
approximately  150  distinct  track  segments  during  the  2  hour  run. 
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6.  Operator  Subjects 


Although  much  of  the  work  preparing  for  this  experiment  centred  on  the  development 
of  a  maritime  simulation,  the  heart  of  the  experiment  is  not  the  simulation.  The 
simulation  is  a  component  of  the  experiment.  It  is  the  interaction  of  the  human  subject 
with  the  simulation  that  is  of  primary  interest.  The  human  subject  plays  an  essential 
role  in  this  experiment.  The  purpose  of  this  experiment  is  to  investigate  how 
additional,  nonorganic  data  affects  an  operator’s  abilities,  and  learning  about  the 
decision-making  processes  employed  by  a  human  operator. 


6.1  Subject  Qualifications 

The  ideal  candidates  for  this  sonar  experiment  are  experienced  naval  sonar  operators. 
The  pool  of  experienced  naval  sonar  operators  is  limited,  however,  and  the  recruitment 
of  a  sufficiently  large  number  of  experienced  operators  to  yield  statistically  relevant 
results  would  be  logistically  challenging.  Since  this  experiment  is  the  first  of  a  series 
of  sonar  experiments,  it  was  deemed  to  be  sufficient  to  use  technically  competent  in- 
house  personnel  as  operators,  provided  that  the  results  of  the  experiment  are 
understood  in  the  correct  context. 

By  using  in-house  personnel  who  are  at  least  familiar  with  acoustics  and/or  have  naval 
military  training,  it  should  be  possible  to  obtain  sufficient  insight  into  the  questions  at 
hand  and  estimate  the  variance  of  the  experimental  results,  so  that  maximal  use  could 
be  made  of  the  group  of  naval  sonar  operators  in  later,  follow-on  experiments. 

Although  this  experiment  makes  use  of  a  simulated  maritime  environment,  that 
environment  is  not  identical  to  the  one  aboard  a  present  day  naval  vessel,  and  the  tools 
and  displays  that  are  used  are  not  the  same  as  those  used  operationally.  This  may  be 
an  impediment  to  an  experienced  operator  who  is  used  to  manually  developing  his  or 
her  own  sonar  tracks  or  making  use  of  sonar  information  that  would  be  available  in  a 
Lofargram  display  but  not  in  the  sonar  track  displays  used  in  this  experiment.  In  order 
to  make  best  use  of  experienced  naval  sonar  personnel,  it  is  important  to  have  a  clear 
idea  as  to  the  number  of  personnel  required,  and  to  have  subject  matter  experts 
evaluate  the  experiment  to  ensure  that  it  is  sufficiently  challenging  to  warrant  the 
additional  logistics  of  bringing  in  a  significant  number  of  these  limited  personnel. 


6.2  Approval  for  the  use  of  Human  Subjects 

Defence  R&D  Canada  (DRDC)  requires  that  all  experiments  that  involve  human 
subjects  be  reviewed  and  approved  by  the  Human  Research  Ethics  Committee  (HREC) 
at  DRDC  Toronto  [5] [15].  The  approval  process,  traditionally  associated  with  medical 
or  physiological  investigations,  requires  the  preparation  of  a  detailed  research  protocol 
and  an  interview  with  the  committee.  The  aim  of  the  HREC  is  to  ensure  that  each 
experiment  is  held  to  a  high  ethical  standard;  that  it  is  designed  to  make  best  use  of  the 
test  subjects,  is  important  enough  to  warrant  use  of  human  subjects,  contributes  to  an 
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approved  program,  and  produces  results  that  cannot  be  obtained  otherwise.  The 
general  requirement  for  review  recognizes  that  the  potential  for  harm  to  the  human 
subjects  is  more  than  just  physical  and  that  maximum  protection  of  subjects  must 
therefore  be  ensured  on  a  broader  basis.  HREC  approval  was  obtained  for  VBE  CA-1 
in  July  2003. 

The  HREC  approved  protocol  for  this  experiment,  Revised  Protocol  #L-416A,  is  in 
Annex  E. 

6.3  Subject  Recruitment,  Care,  Handling  and  Remuneration 

Volunteer  subjects  are  to  be  recruited  for  this  experiment  through  the  use  of  the 
HREC-approved  poster  shown  in  Appendix  A  of  Annex  E.  The  subject  time 
requirement  is  7  hours  in  a  single  day.  This  includes  training,  morning  and  afternoon 
experimentation  sessions  and  a  debriefing  with  breaks  and  lunch  time  in  between. 

Voluntary  consent  forms  and  copies  of  the  experimental  protocol  are  provided  to  the 
subjects  during  the  registration  period.  The  consent  forms  require  a  witnessed 
signature  as  well  as  the  signed  approval  of  the  Section  Head  or  commanding  officer  of 
the  subject.  During  the  training  session,  the  subject  is  instructed  in  the  use  of  Horizon 
and  provided  an  opportunity  to  interact  with  Horizon  prior  to  the  start  of  the 
experimentation  sessions. 

The  morning  and  afternoon  experimentation  sessions  are  independent.  During  one 
session  the  subject  will  operate  the  Coalition  Subject  Station,  during  the  other  the 
subject  will  operate  the  Solo  Subject  Station.  The  debriefing  session  at  the  end  of  the 
day  will  provide  an  opportunity  for  the  subject  to  respond  to  a  questionnaire  as  well  as 
providing  free-form  feedback  on  the  experiment.  The  Run  Director  will  also  be 
present  during  the  entire  experiment  to  respond  to  questions  and  collect  feedback  from 
the  subjects. 

The  level  of  confidence  in  the  accuracy  of  the  results  of  this  experiment  will  depend 
on  the  variance  of  the  results  and  the  number  of  subjects  completing  experimentation 
runs.  Since  the  variance  of  the  results  cannot  be  known  in  advance  and  cannot  be 
reliably  estimated  using  previous  runs  of  the  same  or  a  similar  experiment,  it  was 
decided  to  use  eight  operators  initially  and  then  use  the  results  of  those  runs  to 
estimate  the  number  of  subjects  required  in  this  and  subsequent  experiments. 

The  subjects  are  to  be  reimbursed  a  total  of  $27.64  for  their  participation  in  the 
experiment  in  accordance  with  the  DRDC  Toronto  stress  allowance  schedule. 
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7.  Experimentation  Procedure 


The  experimentation  sessions  involving  human  subjects  are  held  in  the  Virtual  Combat 
Systems  lab  at  DRDC  Atlantic  under  the  direction  of  a  designated  Run  Director.  For 
these  sessions,  the  experimental  apparatus  in  that  lab  is  arranged  as  shown  in  Figure  7. 


7.1  Test  Sessions 

Prior  to  the  use  of  the  volunteer  subjects  as  operators,  test  runs  of  the  experiment  are 
used  to  validate  the  experimental  process  and  infrastructure.  The  test  runs  are  similar 
to  the  actual  experimentation  but  typically  included  only  those  aspects  of  the 
experiment  that  still  required  validation.  The  training,  for  example,  did  not  require 
significant  modification  and  therefore  only  needed  to  be  tested  once.  As  well,  since  a 
member  of  the  development  team  normally  executed  the  test  sessions,  repeated 
training  sessions  were  unnecessary  once  the  training  script  had  been  established.  The 
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Figure  7.  Physical  layout  and  distribution  of  the  experimentation  infrastructure.  Each  dashed  box 
represents  a  single  computer.  All  subject  displays  are  individual  instances  of  Horizon. 
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test  sessions  use  the  designated  test  scenario,  Tl,  that  had  been  set  aside  for  testing  and 
demonstrations. 

Several  minor  changes  were  made  to  address  problems  identified  during  testing 
sessions.  Changes  were  made  to  the  way  in  which  both  the  state  of  the  simulation  and 
the  responses  of  the  operators  to  the  simulation  were  recorded.  Although  the  type, 
position  and  movement  of  the  vessel  traffic  are  predetermined  in  each  scenario,  the 
random  fluctuations  of  the  acoustic  environment  are  not.  Since  the  characteristics  of 
the  sonar  track  segments  presented  to  the  operator  are  dependent  on  the  acoustic 
environment,  these  also  vary  among  multiple  runs  of  the  same  scenario.  To  facilitate 
the  validation  of  the  sonar  and  other  federates  and  the  analysis  of  the  results  of  the 
experiment,  additional  fields  were  added  to  the  log  file  generated  by  the  sonar 
federate.  Other  logging  devices,  made  redundant  by  the  enhanced  sonar  log  format, 
were  removed.  A  description  of  the  final  sonar  federate  log  file  format  is  presented  in 
Annex  D. 

The  log  files  produced  by  each  instance  of  Horizon  record  the  local  system  time,  not 
the  simulation  or  federation  time.  In  order  to  ensure  that  the  local  system  time  on  all 
of  the  computers  is  synchronized,  a  requirement  was  added  for  the  Run  Director  to 
execute  JSS  Clock  Sync,  a  time  synchronization  utility,  prior  to  each  experimentation 
session. 

The  simulation  time  rate  during  the  test  sessions  fell  far  short  of  the  real-time 
performance  originally  expected.  In  order  to  bring  the  response  closer  to  real-time,  it 
was  necessary  to  remove  all  software  that  was  not  essential  to  the  operation  of  the 
simulation.  This  included  Camtasia,  the  Virtual  Maritime  System  Simulation  Display 
(VMSSD)  federate  and  our  own  VBE  logger  federate.  Changes  were  also  made  to  the 
way  that  the  sonar  federate  waited  between  activity  cycles.  A  similar  modification 
was  proposed  for  the  other  VMSA  federates,  some  of  which  had  been  provided  by 
external  sources  [4]  [12]  [20]. 

To  improve  performance,  it  was  also  necessary  to  adjust  the  way  in  which  multiple 
instances  of  Horizon  were  initiated  on  the  operator  stations.  Previous  implementations 
of  Horizon  ran  no  more  than  one  instance  per  computer  and  handled  no  more  than  a 
few  tracks  at  a  time.  This  experiment  typically  produces  at  least  ten  vessel  tracks  and 
several  hundred  sonar  tracks. 

These  problems  did  not  become  apparent  until  the  simulation  was  run  for  extended 
periods  under  heavy  load.  Changes  to  the  infrastructure  configuration  should  not  be 
made  without  further  testing  in  order  to  ensure  that  the  system  will  operate  reliably 
when  the  subject  operators  are  present. 

7.2  Conduct  of  the  Experimentation  Sessions 

The  Run  Director  is  responsible  for  the  execution  of  the  experiment.  In  this  role  he  or 
she  provides  background  materials  to  the  subjects,  ensures  that  their  consent  forms  are 
correctly  completed,  reads  the  training  script  and  demonstrates  the  subject  user 
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interfaces,  responds  to  subject  questions  during  the  experiment,  performs  the  exit 
interviews  and  documents  the  subjects'  responses  on  the  exit  interview  form.  The  Run 
Director  is  also  responsible  for  starting,  ending  and  monitoring  the  simulation  system 
and,  if  necessary,  adjusting  the  experimentation  schedule  to  ensure  the  timely 
acquisition  of  accurate  and  reliable  experimental  data.  Following  each 
experimentation  session,  the  Run  Director  is  responsible  for  archiving  the  session  log 
files. 

The  typical  experimentation  day  schedule  is  shown  in  Table  3.  The  simulation  can  be 
started  by  following  the  first  six  steps  shown  in  Annex  A.  The  batch  file  used  to 
initiate  the  simulation  is  also  shown  in  Annex  A.  The  training  scenario,  which  can  be 
run  for  up  to  120  minutes,  should  be  started  about  20  minutes  prior  to  the  training 
session  in  order  to  provide  a  sufficient  number  of  track  segments  at  the  start  of  the 
training  session.  The  morning  and  afternoon  experimentation  sessions  should  not  be 
started  until  the  operators  are  prepared  to  begin. 


Table  3.  Typical  subject  session  schedule 


TIME 

ACTIVITY 

0900  -  0945 

Subject  registration  and  training 

0945-  1000 

Morning  break 

1000-  1200 

Morning  experimentation  session 

1200-  1300 

Lunch  break 

1300-  1500 

Afternoon  experimentation  session 

1500-  1515 

Afternoon  break 

1515-1600 

Exit  interview 

All  of  the  scenarios  are  designed  to  finish  in  120  minutes.  By  comparing  the 
federation  time  shown  on  the  VMSEM  display  on  the  Run  Director  Station  with  local 
system  time,  the  Run  Director  can  determine  the  progress  of  the  simulation.  The 
VMSEM  display  is  shown  in  Figure  8.  Should  a  scenario  be  slightly  delayed,  it  is 
preferable  to  adjust  the  day’s  schedule  to  accommodate  it,  rather  than  dismissing  the 
operators  prior  to  the  end  of  the  scenario. 

After  the  subjects  have  been  provided  with  copies  of  the  voluntary  consent  forms  and 
other  background  material,  the  Run  Director  should  proceed  with  the  training  session 
using  the  script  shown  in  Annex  B.  Use  of  the  training  script  ensures  that  the  subjects 
are  trained  consistently  and  that  all  relevant  issues  are  addressed.  During  the  training 
session,  the  script  should  be  read  while  the  items  being  described  are 
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-ini  xi 

Federation  Name :  VBE_CA-1 
Scenario  Description :  scenario_description 
Random  Number  Seed :  -1 
Desired  Federation  Rate : 

Actual  Federation  Rate : 

Loop  Index :  1  Max  Loops :  100 

Federate  Time :  0.0  End  Time :  12000.0 


End  Iteration 

End  Federation 

Figure  8.  The  Virtual  Maritime  System  Execution  Manager  Interface  appears  on  the  Run  Director 
Station.  Federation  Time  will  begin  to  increment  shortly  after  all  of  the  indicated  federates 
have  joined  the  federation. 
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demonstrated.  After  the  demonstrations  the  subjects  should  be  given  the  opportunity 
to  interact  with  the  simulation. 

To  protect  subject  confidentiality,  each  subject  will  be  assigned  a  random  number 
between  0  and  99  by  which  their  results  will  be  recorded.  The  Run  Director  will 
record  the  number  assignments,  which  are  to  be  protected,  as  well  as  keeping  a 
separate  list  of  previously  assigned  numbers.  References  to  session  results  will  be  by 
number  only. 

During  the  break  between  the  training  session  and  the  morning  experimentation 
session,  the  Run  Director  will  terminate  the  training  scenario  and  prepare  either 
scenario  1  or  scenario  4,  with  equal  probability,  for  the  morning  session.  The  morning 
scenario  should  not  be  initiated  until  the  subjects  are  ready  to  begin.  The  morning 
session  will  end  when  the  experimentation  scenario  terminates.  Following  the 
morning  session,  the  Run  Director  will  preserve  the  results  of  the  session  by  following 
step  10,  the  final  step  shown,  in  Annex  A.  The  afternoon  session  should  be  run  in  a 
similar  manner,  using  a  different  scenario  than  that  used  in  the  morning  session.  An 
example  of  a  suitable  sequence  of  subject  positions  and  scenarios  is  shown  in  Table  4. 


Table  4.  This  is  an  example  of  a  suitable  scenario  and  position  sequence.  The  subjects  must 
alternate  between  the  multiple  scenarios  and  between  the  coalition  and  solo  positions. 


SUBJECT 

SOLO 

COALITION 

POSITION 

NUMBER 

SCENARIO 

SCENARIO 

SEQUENCE 

1 

1 

4 

cs 

2 

4 

1 

sc 

3 

1 

4 

cs 

4 

4 

1 

sc 

5 

1 

4 

cs 

6 

4 

1 

sc 

7 

1 

4 

cs 

8 

4 

1 

sc 

During  each  experimentation  session,  the  Run  Director  should  observe  the  progress  of 
the  simulation  and  of  the  operators,  and  respond  to  any  questions  the  operators  may 
have.  The  Run  Director  should  also  ensure  that  the  operators  use  only  those  interface 
tools  discussed  and  permitted  in  the  training  session. 

When  the  operators  return  from  their  afternoon  break,  they  should  be  separately 
debriefed  using  the  questions  shown  in  Annex  C.  Before  the  subjects  leave,  the  Run 
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Director  should  ensure  that  their  voluntary  consent  forms  have  been  correctly  filled  out 
and  countersigned.  Accurate  contact  information  is  also  necessary  in  order  for  the 
subjects  to  be  reimbursed  for  their  time. 
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8.  Data  Analysis  Plan 


Following  the  execution  of  the  experimental  sessions,  the  log  files  produced  by  the 
sonar  and  Horizon  federates  and  the  EnterReason  popup  query,  along  with  the  results 
of  the  exit  interviews  and  other  subject  and  Run  Director  feedback  are  to  be  analyzed. 
The  metrics  to  be  used,  both  quantitative  and  qualitative,  are  described  in  the 
following  section. 

8.1  Analysis  Criteria 

8.1 .1  Criteria  for  Quantitative  Analysis 

The  first  objective  of  this  experiment  is  a  comparison  of  two  similar  scenarios  with 
and  without  the  sharing  of  sonar  data.  A  set  of  quantitative  metrics  is  appropriate  in 
this  case  in  order  to  make  clear  and  unambiguous  comparisons  between  the  results  of 
the  scenarios.  The  nature  of  this  aspect  of  the  experiment  lends  itself  well  to 
quantitative  metrics  as  well,  since  the  operator’s  task,  the  development  of  an 
operational  picture  from  primitive  sonar  track  segments,  requires  distinct  and 
deliberate  choices  on  the  part  of  the  operator.  The  metrics  chosen  and  their  derivation 
are  described  in  the  following  section. 

The  metrics  chosen  to  parameterize  the  level  of  operational  picture  development  are 
similar,  but  not  necessarily  identical,  to  the  metrics  developed  for  use  in  other  VBEs 
[8].  This  larger  set  of  metrics  was  collated  and  documented  by  DSTO  and  is  largely 
based  on  the  structure  developed  for  the  Single  Integrated  Air  Picture  Project  (SIAP) 
to  characterize  picture  quality  [22].  Of  particular  interest  in  this  sonar-only  track 
association  case  are  metrics  related  to  picture  clarity,  track  continuity,  and  track 
association,  correctness,  completeness  and  timeliness. 

The  metrics  chosen  are  focussed  on  aspects  understood  to  depend  on  the  operator’s 
level  of  comprehension.  The  task  of  the  operator  in  this  experiment  is  to  identify  sonar 
track  segments  that  were  believed  to  have  originated  from  the  same  vessel  and 
associate  them  into  fused  tracks.  Clearly  the  comprehension  of  the  operator  will  be 
reflected  in  the  speed  and  accuracy  with  which  those  associations  are  formed  as  well 
as  the  completeness  of  the  set  of  fused  tracks. 

Following  the  DSTO  example,  we  define  three  types  of  tracks  in  this  experiment: 

1.  A  primitive  track,  also  called  a  sonar  track  segment  in  this  experiment,  is 
developed  directly  by  the  sonar  federate  from  sonar  data  and  has  no  component 
tracks. 

2.  A  composite  track  is  a  fused  track  made  up  of  one  or  more  primitive  tracks  and/or 
other  composite  tracks.  Composite  tracks  are  produced  when  the  sonar  operator 
uses  the  Active  Fusion  tool  in  Horizon  to  associate  tracks. 
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3.  A  constructed  track  is  a  composite  track  that  is  not  a  component  of  any  other 
composite  track  in  the  operational  picture.  It  is  a  top-level  track  such  as  might  be 
used  for  TMA. 

It  should  be  noted  that,  in  this  experiment,  the  sonar  federates  will  produce  at  least  one 
primitive  track  from  each  observed  vessel  during  each  experimental  session. 

Using  these  track  types  we  define  the  quantitative  metrics  as  follows. 

8. 1. 1 . 1  Picture  Clarity 

Picture  Clarity  measures  the  number  of  constructed  tracks  relative  to  the  number  of 
primitive  tracks. 


„  .  number  of  primitive  tracks 

Picture  Clarity  = - - - 

number  of  constructed  tracks 

Picture  Clarity  cannot  decrease  with  the  number  of  track  associations  regardless  of 
whether  the  associations  are  correct  or  not  since  each  association  removes  one  or  more 
primitive  and/or  composite  tracks  from  the  operational  picture  display  and  replaces 
them  with  a  single  constructed  track.  This  metric  is  only  valid  for  comparisons  at  a 
particular  time,  between  pictures  developed  from  similar  scenarios.  Comparisons 
should  not  be  made  between  applications  of  this  metric  at  differing  times  since, 
without  operator  intervention  and  with  or  without  the  generation  of  additional 
primitive  tracks,  picture  clarity  cannot  decrease  over  time.  This  value  does  not  reflect 
the  accuracy  of  the  track  associations. 

8. 1. 1.2  Track  Continuity 

Track  Continuity  measures  the  number  of  constructed  tracks  relative  to  the  number  of 
observed  vessels.  In  this  experiment,  the  number  of  observed  vessels  was  the  total 
number  of  vessels  less  one,  for  the  observing  vessel,  since  the  sonar  federate  produced 
at  least  one  primitive  track  from  each  observed  vessel. 

_  ,  _  .  .  number  of  observed  vessels 

Track  Continuity  = - 

number  of  constructed  tracks 

Track  Continuity  measures  the  average  number  of  times  that  the  track  of  each 
observed  vessel  appeared  in  the  operational  picture.  In  the  ideally  associated 
operational  picture  there  would  be  only  one  track  per  observed  vessel  and  this  value 
would  therefore  be  unity.  The  Track  Continuity  value  decreases  as  those  ideal  tracks 
become  more  fractured.  Since  the  accuracy  of  a  target  localization  solution  is  related 
to  the  length  and  continuity  of  the  track  being  analyzed,  TMA  is,  in  general,  best 
supported  by  long,  continuous  sonar  tracks.  This  value  does  not  reflect  the  accuracy  of 
the  track  associations. 
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8.1. 1.3  Association  Continuity 


Association  Continuity  measures  the  number  of  disassociations  made  by  the  operator 
relative  to  the  number  of  associations  made  by  the  operator. 

.  .  .  _  .  .  number  of  disassociations  made 

Association  Continuity  = - 

number  of  associations  made 

Association  Continuity  can  be  used  to  compare  the  relative  confidence  levels  of 
operators  within  a  given  scenario  or  the  relative  complexity  of  scenarios  developed  by 
a  given  operator.  It  is  not  necessarily  related  to  the  number  of  constructed  tracks  as  it 
may  include  pairs  of  primitive  tracks  that  are  repeatedly  associated  and  disassociated. 
This  value  does  not  reflect  the  accuracy  of  the  track  associations. 

8.1. 1.4  Association  Correctness 

Association  Correctness  measures  the  number  of  correct  associations  made  relative  to 
the  total  number  of  associations  made. 


.  .  .  _  number  of  correct  associations  made 

Association  Correctness  = - 

number  of  associations  made 

Association  Correctness  can  be  used  to  evaluate  the  accuracy  with  which  an  operator 
makes  track  associations.  Associations  that  are  subsequently  disassociated  are 
considered  to  be  corrections  and  are  not  included  in  this  calculation. 

8.1. 1.5  Association  Completeness 

Association  Completeness  measures  the  number  of  correct  associations  made  versus 
the  total  number  of  possible  correct  associations. 

.  .  .  _  ,  number  of  correct  associations  made 

Association  Completeness  = - 

number  of  possible  correct  associations 

Association  Completeness  can  be  used  to  evaluate  the  completeness  of  the  operational 
picture  developed  by  the  operator.  This  metric  is  calculated  assuming  that  tracks  are 
associated  in  pairs  so  that  the  number  of  possible  associations  is  equal  to  the  number 
of  primitive  tracks  less  the  number  of  observable  vessels.  Associations  that  are 
subsequently  disassociated  are  considered  to  be  corrections  and  are  not  included  in  this 
calculation. 

8. 1.1. 6  Association  Delay 

Association  Delay  measures  the  time  lag  between  the  initiation  of  a  primitive  sonar 
track  and  its  inclusion  in  a  composite  track.  If  multiple  primitive  tracks  are  associated, 
the  latest  initiation  time  is  used.  If  one  or  more  composite  tracks  are  included,  the 
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track  is  decomposed  into  primitive  tracks  and  the  latest  initiation  time  is  used.  The 
Association  Delay  is  the  sum  of  the  time  required  for  sufficient  information  to  be 
available  with  which  to  make  a  decision,  and  the  decision  response  time  of  the 
operator.  The  time  required  for  sufficient  information  to  become  available  may  be 
quite  significant,  especially  in  complex  scenarios  where  multiple  observable  targets 
appear  at  similar  bearings.  Either  component  can  be  factored  out  by  comparing  cases 
using  either  the  same  scenario  or  the  same  operator. 

8. 1. 1 . 7  Measured  Simulation  Time  Rate 

The  Measured  Simulation  Time  Rate  is  the  rate  at  which  simulation  time  advances 
relative  to  real  time. 


„  t  _  elapsed  simulation  time 

Measured  Simulation  Time  Rate  =  — - - 

elapsed  real  time 

This  value,  which  is  measured  from  the  start  of  the  simulation,  is  a  long-term  average 
while  the  Actual  Federation  Rate  shown  on  the  VMSEM  display  is  a  short  term 
average.  In  the  VMSA  this  value  would  ideally  be  equal  to  the  Desired  Federation 
Rate  but,  due  to  the  limitations  of  the  simulation  infrastructure,  is  usually  somewhat 
less. 

8.1 .2  Criteria  for  Qualitative  Analysis 

The  second  objective  of  the  experiment  is  aimed  at  uncovering  the  rationale  used  by  a 
sonar  operator  in  deciding  whether  to  associate  or  disassociate  groups  of  sonar  tracks. 
The  primary  tool  used  to  address  this  objective  is  a  popup  query  that  appears  whenever 
the  track  association  tool  is  used  to  associate  or  disassociate  tracks.  The  exit 
interview,  which  uses  the  subject  debriefing  form  shown  in  Annex  C,  offers  a  second 
opportunity  to  address  this  objective. 

In  both  the  popup  query  and  the  exit  interview,  the  operator  is  presented  with  the 
opportunity  to  make  a  freeform  response  identifying  aspects  of  the  situation  that  the 
operator  identifies  as  being  significant  to  the  decision  process.  As  this  is  necessarily 
subjective,  its  analysis  is  not  well  addressed  through  the  use  of  quantitative  metrics.  A 
catalogue  and  summary  of  the  operator  responses  will  be  used.  The  query  popup  is  not 
entirely  freeform,  as  it  also  offered  the  subject  a  number  of  prepared  responses  from 
which  to  choose.  The  distribution  of  these  choices  is  also  of  interest. 
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9.  Data  Collection  Plan 


9.1  Data  Requirements 

Implementation  of  the  Data  Analysis  Plan  described  in  the  previous  chapter  will 
require  that  particular  pieces  of  data  be  collected  during  each  experimentation  session. 
Some  of  this  data,  which  is  generic  with  respect  to  operator  but  particular  to  the 
session,  can  be  collected  at  the  Run  Director  Station,  while  other  data  is  particular  to 
the  operator  and  should  be  collected  at  each  operator  station. 

Each  experimentation  session  will  execute  one  of  the  prepared  scenarios.  It  is 
essential  that  the  Run  Director  record  the  number  of  the  scenario  used  for  each  session. 

The  sonar  federate  produces  a  text  format  log  file.  Each  instance  of  Horizon  produces 
two  log  files,  one  in  text  format  and  the  other  in  binary  format.  The  format  of  each  of 
the  text  format  files  is  shown  in  Annex  D. 

9.1.1  Picture  Clarity 

The  requirements  of  the  Picture  Clarity  metric  can  be  met  by  extracting  the  number  of 
primitive  tracks  from  the  sonar  log  file  and  the  number  of  constructed  tracks  from  each 
Horizon  log  file.  Horizon  names  each  track  from  a  particular  sensor  consecutively. 

As  a  result,  the  number  of  primitive  tracks  from  a  sonar  is  equal  to  the  highest  primary 
track  number  from  that  sonar. 

The  number  of  constructed  tracks  is  not  necessarily  equal  to  the  value  of  the  highest 
composite  track  number  since  a  constructed  track  can  include  composite  tracks.  For 
example  as  shown  in  Figure  9,  if  composite  track  FI  is  made  up  of  primitive  tracks  PI 
and  P2,  then  FI  will  be  a  constructed  track  as  long  as  FI  is  not  disassociated  and  is  not 
a  component  of  any  other  composite  track.  If  composite  track  FI  is  associated  with 
primitive  track  P3  to  form  composite  track  F2,  then  track  FI  will  no  longer  be  a 
component  of  any  other  composite  track.  If  composite  track  FI  is  associated  with 


F2 
/  \ 
FI  P3 
/  \ 

PI  P2 


Figure  9.  Track  nomenclature.  PI,  P2  and  P3  are  primitive  tracks,  and  FI  and  F2  are  composite 
tracks,  but  only  F2  is  a  constructed  track  because  it  is  not  a  component  of  a  composite 
track. 
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primitive  track  P3  to  form  composite  track  F2,  then  track  FI  will  no  longer  be  a 
constructed  track  since  it  is  a  component  of  composite  track  F2.  F2  would  then  be  a 
constructed  track  so  long  as  it  is  not  disassociated  and  is  not  a  component  of  any  other 
composite  track.  The  association  and  disassociation  of  tracks  is  recorded  in  the 
Horizon  log  files. 

9.1.2  Track  Continuity 

The  requirements  of  the  Track  Continuity  metric  can  be  met  by  extracting  the  number 
of  observed  vessels  from  the  sonar  log  file  or  the  scenario  file,  and  the  number  of 
constructed  tracks  from  the  Horizon  log  file. 

9.1.3  Association  Continuity 

The  requirements  of  the  Association  Continuity  metric  can  be  met  by  extracting  the 
number  of  associations  and  disassociations  made  from  the  Horizon  log  file.  Although 
association  events  are  not  recorded  in  the  Horizon  text  log  file,  the  subsequent 
appearance  of  a  new  composite  track,  specified  with  its  component  tracks,  can  be  used 
to  reconstruct  the  association  decision.  Similarly,  the  logging  of  a  track  that  had 
previously  been  a  component  of  a  composite  track  can  be  used  to  reconstruct  a 
disassociation  decision. 

9.1.4  Association  Correctness 

The  requirements  of  the  Association  Correctness  metric  can  be  met  by  extracting  the 
number  of  associations  and  the  components  of  each  composite  track  from  the  Horizon 
log  file.  The  number  of  correct  associations  can  be  determined  by  relating  each 
primitive  track  to  the  vessel  or  vessels  from  which  it  was  produced  and  ensuring  that 
these  vessels  are  consistent.  The  relationship  between  the  primitive  track  segments 
and  the  source  vessels  can  be  found  in  the  sonar  log  file.  This  may  be  ambiguous 
when  multiple  vessels  at  a  similar  bearing  produce  a  single  primitive  track 
representing  them  as  a  group.  An  additional  test  of  a  correct  association  is  that  all  of 
the  component  tracks  must  be  consecutive  in  time. 

9.1.5  Association  Completeness 

The  requirements  of  the  Association  Completeness  metric  can  be  met  by  extracting  the 
number  of  possible  correct  associations  from  the  sonar  log  file.  The  number  of 
possible  correct  associations  can  be  developed  from  the  number  of  primitive  tracks 
relating  to  the  vessel  in  question.  The  number  of  correct  associations  can  be 
determined  as  described  in  the  previous  paragraph. 

9.1.6  Association  Delay 

The  requirements  of  the  Association  Delay  metric  can  be  met  by  extracting  the 
association  time  of  each  composite  track  from  the  Horizon  log  file  and  the  initiation 
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time  of  each  of  its  primitive  track  components  from  the  sonar  log  file.  Those 
components  that  are  composite  tracks  should  be  further  reduced  to  primitive  tracks 
before  an  evaluation  is  made. 


9.1.7  Measured  Simulation  Time  Rate 

The  requirements  for  the  Measured  Simulation  Time  Rate  can  be  met  by  extracting  the 
initiation  times  of  the  primary  tracks  from  both  the  sonar  log  file  and  the  Horizon  log 
file.  The  sonar  log  is  recorded  in  federation  time  while  the  Horizon  log  is  recorded  in 
system  time.  Elapsed  time  in  each  time-space  can  be  determined  by  the  difference 
between  the  initiation  time  of  the  first  primary  track  and  the  most  recently  initiated 
primary  track.  If  the  simulation  exhibits  slower  than  real-time  performance,  the 
elapsed  federation  time  will  be  increasingly  less  than  the  elapsed  system  time. 

9.1 .8  Requirements  for  Qualitative  Analysis 

The  EnterReason  popup  query  produces  a  log  text  file.  The  file  contains  a  numbered 
list  of  possible  rationales  for  an  association  or  disassociation  decision  followed  by  the 
system  times  at  which  each  decision  was  made,  the  type  of  decision,  and  the  decision 
rationale  offered  by  the  operator. 

The  requirement  for  Qualitative  Analysis  can  be  met  by  compiling  and  analyzing  the 
EnterReason  log  file,  the  results  of  the  exit  interview,  and  other  comments  made  by  the 
operators. 

9.2  Recording  the  Results  of  an  Experimentation  Session 

The  Run  Director  is  responsible  for  adequately  preserving  the  results  of  each 
experimentation  session.  This  is  done  by  following  the  final  step  in  the 
Experimentation  Software  Setup  procedure  shown  in  Annex  A,  which  will  copy  the 
sonar,  Horizon  and  EnterReason  log  files  to  a  unique  directory  on  the  shared  drive  of 
the  Run  Director  Station.  The  Run  Director  must  also  record  the  station  used,  either 
solo  or  coalition,  and  the  scenario  seen  by  each  operator.  The  operators  should  be 
indicated  by  subject  number  only,  not  by  name. 

The  Run  Director  is  responsible  for  maintaining  a  backup  copy  of  the  shared  drive 
containing  the  experimentation  software  and  the  results  to  date.  This  is  facilitated  by 
good  experiment  design,  as  in  the  current  experimentation  configuration,  where  all  of 
this  material  is  on  drive  v:  of  the  Run  Director  Station,  shared  to  the  other  stations,  and 
then  remotely  mounted  on  each  station  as  drive  v. 

It  is  not  strictly  necessary  that  sufficient  information  be  preserved  to  exactly  repeat  the 
experiment.  To  do  so  would  be  especially  challenging  since  the  variability  of  the 
sonar  tracks  is  due  in  part  to  the  influence  of  noise  calculated  using  a  random  number 
generator.  It  is  sufficient  that  enough  information  be  preserved  to  permit  later  testing 
for  correct  operation  of  the  experiment  from  the  perspective  of  the  operator.  This  can 
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be  done  while  analysing  the  results  of  the  experiment.  A  successful  experiment  is  one 
in  which  the  experimental  objectives  are  met  and  the  experimentation  sessions  proceed 
as  described  in  the  conceptual  model. 
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10.  Distribution  of  the  Experimental  Results 


Successful  completion  of  this  experiment  will  include  the  production  of  a  number  of 
documents  to  describe  the  context,  process,  operation,  results  and  conclusions  of  the 
experiment. 

1.  This  experimental  plan  is  intended  to  provide  some  context  for  the  experiment  as 
well  as  a  blueprint  and  guide  for  the  implementation  and  execution  of  the  VBE 
CA-1  experiment.  It  will  be  published  as  a  DRDC  Atlantic  Technical 
Memorandum. 

2.  An  HREC  protocol  document  was  produced  as  part  of  the  HREC  review  process. 

A  copy  of  the  protocol  as  approved  is  included  as  Annex  E  to  this  experimental 
plan.  Following  the  completion  of  this  experiment,  a  letter  will  be  sent  to  the 
HREC  indicating  the  number  of  operator  subjects  that  would  be  required  for  a 
follow-on  experiment  involving  experienced  naval  sonar  operators.  The  voluntary 
consent  forms  obtained  in  this  experiment  must  also  be  submitted  to  the  HREC. 

3.  A  pair  of  DRDC  Atlantic  Technical  Memoranda  will  be  produced  following  the 
experiment  to  describe  it  from  both  a  scientific  perspective  and  an  implementation 
perspective.  The  scientific  paper  will  discuss  the  motivation  and  rationale  for  the 
experiment,  and  present  an  analysis  of  the  results  along  with  conclusions  pertinent 
to  the  experimental  objectives.  The  implementation  paper  will  focus  on  the 
mechanics  of  the  experiment  and  is  intended  to  be  of  greatest  interest  to  personnel 
involved  in  the  performance  and  execution  of  experiments  using  a  similar 
experimentation  infrastructure.  These  papers  will  constitute  the  primary  record  of 
the  experiment. 

4.  A  number  of  shorter  papers  and  presentations  will  present  a  synopsis  of  the 
experiment  to  external  audiences  from  a  perspective  appropriate  to  the  target 
audience.  These  include  presentations  and  papers  to  TTCP  MAR  TP-1  and  The  9th 
International  Command  and  Control  Research  and  Technology  Symposium. 
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Annex  A  Experiment  Software  Setup 

To  run  the  experiment  using  the  experimentation  configuration  described  here: 

1.  Log  into  the  computers  Run  Director  (Archer),  Operator  1  Ownship  (Cherokee), 
Operator  1  Allied  (Seneca)  and  Operator  2  (Piper).  Use  the  vbe  user  account  on  the 
Operator  machines. 

2.  On  each  of  the  computers,  click  on  the  Synchronize  button  in  the  JSS  Clock  Sync 
window.  Then  click  on  the  Exit  button  in  the  same  window  to  close  this  application. 

3.  Select  the  scenario  to  be  run  by  editing  V:\VBE  CA-l\VMSA\bin\Start  Gamboard.bat. 
The  full  path  name  of  the  scenario  file  must  be  shown  in  quotes  immediately  after  the 
word  “Gameboard”  on  the  line  beginning  with  “Gameboard”. 

4.  On  the  Run  Director  computer,  execute  the  batch  file  Run  Director  -  2  operators.  On 
the  Operator  1  Ownship  computer,  execute  the  batch  file  Operator  1  -  Ownship.  On 
the  Operator  1  Allied  computer,  execute  the  batch  file  Operator  1  -  Allied  ship.  On 
the  Operator  2  computer,  execute  the  batch  file  Operator  2.  In  all  cases,  the  batch 
files  can  be  found  on  the  desktop  or  in  the  V:\VBE_CA-l\bin  folder. 

5.  The  execution  manager,  VMSEM,  will  appear  on  the  Run  Director  display.  The 
federation  will  not  start  until  about  two  minutes  after  all  of  the  federates  show  True  in 
the  Joined  column.  Experiment  progress,  including  federation  time  in  seconds,  is 
shown  on  the  Execution  Manager. 

6.  The  Operator  1  Allied  computer  represents  the  allied  vessel.  Maximize  the  Horizon 
window,  then  click  on  the  TBRG  button  to  select  the  time-bearing  display.  Click  on 
the  30  button  to  set  the  display  to  show  30  minutes  of  data.  Maximize  the  Enter 
Reason  application,  center  it  in  the  track  display  and  then  minimize  it. 

7.  The  Operator  1  Ownship  and  Operator  2  computers  represent  the  ownship.  On  each 
computer,  drag  the  Horizon  display  without  the  active  fusion  plug-in  onto  the 
rightmost  monitor,  maximize  it  then  click  on  the  MAP  button  to  select  the  chart 
display.  Click  on  the  Ownship  button  to  center  the  display  on  the  ownship  and  on  the 
100K  button  to  show  a  100  km  ring  around  the  ownship.  Maximize  the  remaining 
Horizon  display  on  the  leftmost  monitor,  then  click  on  the  TBRG  button  to  select  the 
time  bearing  display.  Click  on  the  30  button  to  set  the  display  to  show  30  minutes  of 
data.  Maximize  the  Enter  Reason  application,  center  it  in  the  track  display  and  then 
minimize  it. 

8.  Seat  the  operators  and  run  the  experiment. 

9.  To  end  the  experiment,  close  each  of  the  program  windows  on  each  computer. 
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10.  Following  the  experiment,  create  a  new  directory  under  the  V:\Logfiles  folder  and 

rename  it  to  reflect  the  date  and  time  of  the  experiment.  Copy  configuration  and  log 

files  into  this  folder  as  follows: 

a.  Copy  the  sonar  log  files  sonar. cfg  and  sonar.log  from  V:\VBE_CA- 
l\VMSA\federates\Sonar\bin  into  the  primary  folder. 

b.  Copy  the  reasons  logging  file  from  C:\reasons\file  on  the  Operator  1  Ownship 
computer  as  reasons _coalition_ownship.txt. 

c.  Copy  the  Horizon  text  logger  file  from  C:\  text  log  Jiles  on  the  Operator  1 
Ownship  computer  as  horizon_coalition_ownship.log. 

d.  Copy  the  Horizon  binary  logger  file  from  C:\  binary  Jog  Jiles  on  the  Operator  1 
Ownship  computer  as  horizon_coalition_ownship.dat. 

e.  Copy  the  reasons  logging  file  from  C:\reasons\file  on  the  Operator  1  Allied 
computer  as  reasons _coalition_allied.txt. 

f.  Copy  the  Horizon  text  logger  file  from  C:\  text  log  Jles  on  the  Operator  1  Allied 
computer  as  horizon_coalition_ownship.log. 

g.  Copy  the  Horizon  binary  logger  file  from  C:  \  binary  Jog  Jles  on  the  Operator  1 
Allied  computer  as  horizon_coalition_allied.dat. 

h.  Copy  the  reasons  logging  file  from  C:\reasons\file  on  the  Operator  2  computer  as 
reasons _solo_ownship.  txt. 

i.  Copy  the  Horizon  text  logger  file  from  C:\  text  Jog  Jles  on  the  Operator  2 
computer  as  horizon_solo_ownship.log. 

j.  Copy  the  Horizon  binary  logger  file  from  C:\  binary  Jog  Jles  on  the  Operator  2 
computer  as  horizon_solo_ownship.dat. 
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@echo  This  script  will  set  up  the  Run  Director's 
position  in  a  2  operator  federation. 

@echo . 

@echo  Press  Ctrl-C  to  exit  now  or 
@pause 

cd  /d  v:\vbe_ca-l\vmsa\bin 
start  /MIN  "RTI "  "Start  RTI.bat" 

start  /MIN  "RTI  Console"  "Start  RTI  console.bat" 

start  /MIN  "VMSEM"  "Start  VMSEM.bat" 
start  /MIN  "Motion"  "Start  Generic  Motion.bat" 
start  /MIN  "Gameboard"  "Start  GameBoard.bat" 
start  /MIN  "Sonar"  "Start  Sonar.bat" 

@rem  start  /MIN  "Horizon  Allied"  "Start 
Horizon_allied . bat" 

@rem  start  /MIN  "Horizon  Own"  "Start  Horizon_own.bat" 

@rem  start  /MIN  "Horizon  0wn2"  "Start  Horizon_own2.bat" 

@rem  start  /MIN  "Horizon  0wn3"  "Start  Horizon_own3.bat" 

@rem  start  /MIN  "Horizon  0wn4"  "Start  Horizon_own4.bat" 

@rem  start  /MIN  "Reasons"  "c:\EnterReasons.exe" 


Figure  10.  This  batch  file,  Run  Director  -  2  Operators,  is  used  at  the  Run  Director  Station  to  initiate 
the  simulation.  Similar,  appropriately  modified  batch  files  are  used  on  the  other  computers. 
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Annex  B  Subject  Training  Scripts 

. m . w . l . 


The  following  points  should  be  covered  with  the  experiment  subjects. 


B.1  Introduction 

In  this  simulation,  you  will  be  acting  as  the  sonar  operator  aboard  a  frigate  that  is  using 
a  towed  array  sonar  to  monitor  vessel  traffic  in  a  narrow  strait.  The  position  of  the 
ownship  in  the  strait  is  shown  in  the  navigational  chart  display  on  the  center  screen. 

An  allied  ship  using  a  similar  towed  array  sonar  may  be  available  to  assist  you  by 
providing  a  copy  of  its  sonar  tracks.  The  position  of  the  allied  vessel  is  also  shown  in 
the  navigational  display.  No  other  information  about  vessel  traffic  in  the  strait  is 
available.  The  task  of  the  operator  is  to  develop  as  complete  of  a  local  operational 
picture  as  possible.  Operational  picture  quality  is  determined  by  the  clarity  of  the 
ownship  track  display. 

The  number  and  course  of  the  vessels  in  the  strait  is  unknown,  although  each  vessel 
has  an  acoustic  signature  that  can  cause  track  segments  to  appear  on  the  passive  sonar 
display.  The  nature  of  the  targets,  the  local  environment  and  the  sonar  system  are  such 
that  the  there  is  only  direct  path  propagation,  i.e.  a  vessel  can  produce  no  more  than 
one  track  segment  at  a  time.  A  track  segment  may  be  terminated  for  a  number  of 
reasons  including  target  range,  high  background  noise  or  interfering  signals.  There  is 
no  restriction  on  the  number  of  times  that  the  track  of  a  vessel  may  be  terminated  and 
reinitiated.  Without  track  association,  the  sonar  system  will  interpret  each  track 
segment  as  a  unique  and  unrelated  piece  of  information. 

Target  motion  analysis  (TMA)  which  can  be  used  to  derive  target  position  estimates 
from  bearing  estimates,  is  strongly  affected  by  the  duration  and  completeness  of  the 
track  being  analyzed.  Therefore,  correctly  associating  multiple  track  segments  into  a 
longer,  more  complete  master  track  will  greatly  improve  the  accuracy  of  the  target 
position  estimate. 

A  tool  is  available  to  associate  track  segments  that  are  believed  to  have  originated 
from  the  same  source.  The  same  tool  can  also  be  used  to  disassociate  fused  tracks 
back  into  their  original  segments.  One  of  the  objectives  of  this  experiment  is  to 
investigate  the  process  by  which  an  operator  decides  to  associate  or  disassociate  tracks 
and  so  a  popup  menu  will  appear  whenever  the  association  tool  is  used.  The  tool  will 
ask  the  operator  to  either  select  a  reason  for  the  decision  from  a  checklist  or  enter  it  in 
a  text  box.  It  is  important  that  your  selection  accurately  represents  the  reasoning  you 
used  to  make  the  association  or  disassociation  decision.  The  text  box  option  should  be 
used  if  you  don’t  see  your  reason  in  the  checklist.  This  is  a  key  part  of  the  experiment. 

During  the  following  description  of  the  simulation  interface  you  are  encouraged  to 
operate  the  tools  and  features  after  they  are  discussed.  Ask  the  Run  Director  if  you 
have  any  questions  about  the  simulation  or  the  controls.  Please  do  not  use  any  controls 
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other  than  those  indicated  by  the  Run  Director.  If  you  would  like  to  make  use  of  a 
feature  that  is  not  available  or  has  not  been  explained,  please  ask  the  Run  Director  for 
assistance  at  any  time. 


B.2  The  Navigational  Chart  Display 

The  navigational  chart  displays  the  navigable  waters  in  the  region  of  interest.  The 
ownship  and  allied  ship  are  shown  here  as  grey  arrows  labelled  Ownship  and 
ASJVessel  respectively.  The  range  rings  are  labelled  in  kilometres.  Yellow  labels  on 
the  periphery  of  the  range  rings  indicate  ownship  track  labels  but  will  not  be  used  in 
this  experiment,  as  they  will  quickly  become  illegible. 

The  Ownship  button  can  be  used  to  center  the  chart  on  the  ownship. 

The  Lat/Long  button  can  be  used  to  center  the  chart  on  an  arbitrary  point  by  clicking 
on  the  button  and  then  on  that  point. 

The  Object  button  can  be  used  to  center  the  chart  on  the  allied  ship  by  clicking  on  the 
button  and  then  on  the  allied  ship. 

The  5K,  10K,  20K,  50K  or  100K  buttons  can  be  used  to  change  the  map  scale. 

Using  a  mouse  button  to  indicate  a  box  on  the  chart  will  cause  it  to  zoom  in  on  that 
box.  Clicking  the  right  mouse  button  on  the  chart  will  cause  it  to  return  to  the  last 
button- selected  scale. 

The  cursor  position  is  shown  in  the  lower  right  corner  as  range,  bearing  from  the 
center  and  as  latitude,  longitude. 

B.3  The  Sonar  Track  Display 

The  sonar  display  shows  sonar  tracks  for  a  specific  ship  in  time-bearing  format.  Each 
sonar  track  segment  is  drawn  as  a  series  of  open  circles  followed  by  a  question  mark. 
The  ship’s  course  is  indicated  as  a  series  of  solid  grey  circles.  Plot  time  is  shown  in 
HH:MM  format  and  bearing  is  shown  in  degrees  absolute. 

The  cursor  position  is  shown  in  the  lower  right  comer  as  time  and  bearing. 

The  Active  Fusion  tool  in  the  lower  left  can  be  used  to  associate  track  segments  into 
fused  tracks.  Begin  by  clicking  on  the  Select  Tracks  button  until  it  is  green  and  the 
Selected  Tracks  window  above  it  is  empty.  Track  segments  in  the  time-bearing  plot 
can  be  selected  by  left-clicking  on  them.  They  will  change  to  bold  and  appear  in  the 
Selected  Tracks  window  when  they  are  selected.  If  the  mouse-click  is  ambiguous,  a 
checklist  of  track  segments  near  the  click  position  will  appear.  Track  segments  that 
are  selected  will  have  a  checkmark  next  to  them.  Highlight  and  click  on  a  segment  to 
toggle  its  selection. 
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Dragging  the  left  mouse  button  across  an  area  of  the  screen  will  cause  the  display  to 
zoom  in  on  the  selected  area.  Clicking  the  right  mouse  button  on  the  display  will 
cause  the  display  to  return  to  full-scale. 

To  associate  the  tracks  shown  in  the  Selected  Tracks  window  into  a  single  fused  track 
using  the  Combine  All  Tracks  method,  click  on  the  green  checkmark  button.  The 
selected  tracks  will  be  removed  from  the  time-bearing  plot  and  replaced  with  a  single 
track  drawn  as  a  series  of  lightning  bolts  followed  by  a  question  mark. 

Whenever  the  green  checkmark  button  is  clicked,  a  popup  window  appears  asking  why 
that  choice  was  made.  To  respond,  click  on  the  button  beside  the  best  response.  If  the 
button  for  the  “other”  response  is  selected,  a  text  box  will  appear  in  which  to  describe 
the  “other”  reason.  Click  on  “OK”  after  a  reason  has  been  entered  and  the  popup 
window  will  disappear. 

To  disassociate  a  fused  track,  select  it  then  click  on  the  red  “X”  button.  The  fused 
track  will  be  removed  from  the  plot  and  the  original  track  segments  will  reappear. 
Clicking  on  the  red  “X”  button  will  also  cause  the  popup  menu  to  appear.  Click  on 
“OK”  after  a  reason  has  been  entered  and  the  popup  window  will  disappear. 

The  Annotation  Generator  tool  in  the  lower  middle  can  be  used  to  annotate  tracks. 
Begin  by  selecting  the  track  to  be  labelled  in  the  box  labelled  Associated  Track.  Only 
those  tracks  shown  on  the  corresponding  plot  above  can  be  annotated.  For  the 
Ownship,  these  track  labels  begin  with  CA_Sonar,  for  the  allied  ship,  these  track  labels 
begin  with  ASJVessel.  Having  selected  a  track,  enter  the  annotation  in  the  box 
labelled  Annotation  Text.  In  the  box  labelled  Age  (seconds)  enter  the  number  of 
seconds  that  have  elapsed  since  the  desired  annotation  time.  When  the  Create  button 
is  clicked,  an  annotation  will  appear  next  to  the  specified  track  at  the  specified  time. 

As  time  advances  and  the  display  scrolls  down,  so  will  the  annotation.  Annotations 
made  to  track  “<none>”  will  appear  on  the  right  side  of  the  time-bearing  plot. 

Annotations  can  be  placed  at  an  arbitrary  location  in  the  time-bearing  plot  by 
specifying  the  coordinates  of  the  location  in  the  Bearing  and  Age  boxes.  Make  certain 
that  the  Bearing  Valid  box  is  checked  when  using  this  option  and  unchecked  when 
locating  by  track. 

The  Annotation  Maintenance  tool  in  the  lower  middle  can  be  used  to  alter  annotations. 
Begin  by  clicking  on  the  Select  Annotations  button  until  it  is  green,  then  left-click  on 
the  annotation  to  be  altered  or  select  it  from  the  drop-down  menu  in  the  box  labelled 
Annotation.  The  annotation  text  can  be  edited  in  the  box  labelled  Text  and  the  track  to 
which  it  is  attached  can  be  altered  in  the  box  labelled  Track.  Click  on  Modify  to 
update  the  annotation  or  Delete  to  delete  it. 

Clicking  on  the  Fused  and  Sonar  buttons  in  the  top  left  corner  will  turn  those  tracks  off 
(yellow)  and  on  (green). 

Clicking  on  the  Notes  button  in  the  top  left  corner  will  turn  track  annotations  off 
(yellow)  and  on  (green). 
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The  time  duration  of  the  display  can  be  selected  using  the  30,  60  and  90  minute 
buttons  on  the  middle  left  side  of  the  display. 

The  center  of  the  bearing  axis  of  the  display  can  be  toggled  between  0°  and  180°  by 
clicking  on  the  Shift  180°  button  on  the  middle  right  side  of  the  display. 

No  other  controls  will  be  needed  in  this  experiment.  Please  ask  the  Run  Director  for 
assistance  before  operating  any  other  controls.  Please  take  some  time  now  to  exercise 
those  controls  that  have  just  been  described. 
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Annex  C  Subject  Debriefing  Form 


Ask  the  subject  the  following  questions  and  record  the  responses  and  any  additional 
comments. 

1 .  How  do  you  know  about  sonar?  How  many  years  of  experience  do  you  have? 

2.  What  is  your  practical  experience  with  sonar  data?  How  many  years  of  experience 
do  you  have? 

3.  How  satisfied  were  you  with  the  local  operating  picture  that  you  developed? 

4.  What  use  did  you  make  of  the  allied  ship  display? 

5.  How  would  you  rank  the  value  of  the  allied  ship  tracks  in  your  development  of  the 
local  operating  picture  (high,  medium,  low)? 

6.  What  additional  controls  or  tools  would  you  have  liked  to  have  had? 

7.  Any  other  comments? 


DRDC  Atlantic  TM  2003-158 


53 


This  page  intentionally  left  blank. 


54 


DRDC  Atlantic  TM  2003-158 


Annex  D  Log  File  Formats 


D.1  Horizon  Log  File  Format 

The  Horizon  log  file  format  is  shown  in  Table  5.  The  data  is  comma  delimited  with  an 
update  interval  of  15  seconds.  A  separate  line  is  required  for  each  track  update. 


Table  5.  Horizon  log  file  format. 


Field 

Type 

Description 

Precision 

Invalid  Value 

System  Time 

String 

System  Time  (hh:mm:ss  AM/PM) 

Data  Time 

String 

Time  at  which  the  data  was 
recorded  (hh:mm:ss  AM/PM) 

Composite 

Tracks 

String 

Zero  or  more  strings  in  the  format 
of  the  Track  Number  field  and 
separated  by  the  7’  character  that 
identify  the  member  tracks  of  an 
association 

Track  Number 

String 

Provides  a  unique  track 
identification  in  the  form  cHorizon 
data  source>:<  track  ID> 

Master  ID 

String 

Track  master  ID  as  set  by  the 
operator,  otherwise  blank. 

Latitude 

Float 

Latitude  in  decimal  degrees 

0.0001 

-999.999 

Longitude 

Float 

Longitude  in  decimal  degrees 

0.0001 

-999.999 

Error_Major 

Float 

Semi-major  axis  of  the  ellipse  that 
forms  the  area  of  uncertainty 

0.1 

-1.0 

Error_Minor 

Float 

Semi-Minor  axis  of  the  ellipse  that 
forms  the  area  of  uncertainty 

0.1 

-1.0 

Error_Angle 

Float 

Degrees  from  vertical  of  the 
ellipse  that  forms  the  area  of 
uncertainty 

0.1 

-1.0 

Course 

Float 

Degrees  [0-360) 

0.1 

-1.0 

Speed 

Float 

m/s 

0.1 

-1.0 

Source  ID 

String 

Name  of  the  source  platform  that 
is  providing  the  track  data 

Source 

Latitude 

Float 

Latitude  in  decimal  degrees  of  the 
source  platform  at  data  time 

0.001 

-999.999 

Source 

Longitude 

Float 

Longitude  in  decimal  degrees  of 
the  source  platform  at  data  time 

0.001 

-999.999 

Bearing 

Float 

Absolute  bearing  in  degrees  [0- 
360)  from  the  source  platform  to 
the  track 

0.1 

-1.0 
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D.2  Sonar  Log  File  Format 


The  Sonar  log  file  format  is  shown  in  Table  6.  The  data  is  tab  delimited  with  an 
specifiable  update  interval,  typically  5  seconds.  A  separate  line  is  required  for  each 
track  update. 


Table  6.  Sonar  log  file  format. 


Field 

Type 

Description 

Precision 

Note 

Type  Code 

Integer 

101  (initiate  new  track),  201 
(update  existing  track),  or  301 
(terminate  track). 

Track  Name 

String 

Name  of  the  source  that  is 
providing  the  track  data. 

Track  Number 

Integer 

Unique  sequential  identification 
number  for  this  track  from  this 

source. 

Time  Step 

Integer 

Federation  time  step  at  which  the 
track  update  occurred 

Bearing 

Float 

Absolute  bearing  of  the  centroid 
of  the  acoustic  source(s) 

(degrees). 

0.0001 

type  codes  101 
and  201  only 

Range 

Float 

Range  of  the  centroid  of  the 
acoustic  source(s)  (km). 

0.0001 

type  codes  101 
and  201  only 

Signal  Excess 

Float 

Signal  excess  of  the  received 
acoustic  signal  (dB). 

0.0001 

type  codes  101 
and  201  only 

Target  Count 

Integer 

Number  of  acoustic  emitters 
represented  by  this  track. 

type  codes  101 
and  201  only 

Target 

Name(s) 

String 

Name(s)  of  the  acoustic  emitters 
represented  by  this  track 

type  codes  101 
and  201  only 
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Annex  E  The  HREC-Approved  Protocol  for  VBE  CA-1 


Protocol:  L-416A 

Title:  Coalition  passive  sonar  track  sharing  and  association 

Principal  Investigator:  Garfield  R.  Mellema,  DRDC  Atlantic 
Co-Investigator:  Tania  E.  Wentzell,  DRDC  Atlantic 

DRDC  Thrust:  llcs/llbk 

Executive  Summary 

The  ability  to  exchange  tactical  information  among  coalition  partners  has  the  potential 
to  significantly  improve  our  ability  to  conduct  underwater  warfare.  The  degree  of 
improvement,  however,  is  dependent  on  the  rate  at  which  the  information  can  be  transferred 
between  partners  as  well  as  their  ability  to  process  the  received  data.  In  a  typical  scenario,  as 
sensor  information  is  processed  into  target  location,  course  and  speed,  the  data  volume 
decreases  dramatically  while  the  transmission  delay  increases.  In  some  cases  it  may  be 
worthwhile  to  trade  off  data  volume  to  minimize  transmission  delay.  Some  of  the  more 
sophisticated  processing  techniques  such  as  cross-correlation  or  triangulation,  which  extract 
time  or  location  information  from  comparisons  of  low  level  data  from  multiple  sensors,  would 
not  be  available  without  this  shared  data. 

In  order  to  benefit  from  shared  data,  the  recipient  must  be  able  to  receive  and  process 
it  to  derive  added  value  with  sufficient  speed  to  justify  the  increased  cost  of  transmitting  it  at  a 
low  level.  Otherwise,  it  would  be  better  to  process  it  at  the  source  and  send  only  the 
processed  data.  Although  data  exchange  issues  that  are  related  to  automated  systems,  such  as 
bandwidth  or  processing  speed,  can  usually  be  clearly  specified,  issues  related  to  the  human 
operator  may  be  more  difficult  to  define.  The  minimum  level  at  which  data  exchange  is 
beneficial  may  be  masked  by  issues  more  related  to  operator  loading  than  to  operator 
comprehension. 

This  experiment  will  investigate  the  value  of  sharing  sonar  data  at  a  low  level  of 
development  between  the  operator’s  own  ship  and  a  geographically  remote  allied  vessel.  The 
outcome  of  this  experiment  will  be  determined  by  measuring  the  quality  of  the  local  operating 
picture  developed  by  the  operator  on  board  the  own  ship. 

A  single  acoustic  source  in  the  local  environment  may  cause  multiple  sonar  track 
segments  to  appear  on  a  sonar  screen,  each  represented  to  the  sonar  operator  as  a  potentially 
unique  target.  These  track  segments  may  be  separated  in  bearing  due  to  differing  propagation 
paths  between  the  source  and  receiver,  and  /  or  separated  in  time  due  to  changes  in  the  source 
level,  source  or  receiver  location  or  the  local  environment.  Multiple  sonar  track  segments  that 
are  believed  to  have  originated  from  the  same  source  can  be  grouped  into  a  master  track  in 
spite  of  their  differences  in  time,  frequency  or  bearing.  This  track  association  process  is  very 
labour  intensive  and  relies  heavily  on  the  sonar  operator’s  training  and  experience. 
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Automation  of  this  task  is  desirable,  as  is  insight  into  the  decision  process  used  by  human 
operators  to  decide  whether  or  not  to  associate  track  segments. 

In  this  experiment,  the  subject  will  be  asked  to  develop  a  local  operational  picture 
indicating  the  number  and  actions  of  nearby  vessels.  The  subject  will  be  supplied  with 
information  about  the  local  environment  and  the  positions  of  the  own  and  allied  ships  as  well 
as  track  information  from  the  own  ship  sonar  display.  Comparisons  will  be  made  between 
local  operational  pictures  developed  with  and  without  the  benefit  of  additional  sonar  track 
information  from  the  allied  ship.  Measurements  will  also  be  made  of  the  number  and  quality 
of  track  association  and  disassociation  decisions  made  by  the  subject  and  the  logic  behind 
those  decisions  as  they  are  made.  The  subjects  will  be  further  debriefed  in  a  follow-up 
interview. 

This  experiment  presents  minimal  risk  to  the  test  subjects.  No  psychological 
measures  will  be  taken  and  no  emotionally  distressing  stimuli  will  be  used.  There  is  a  risk  of 
eyestrain  and  fatigue  associated  with  viewing  a  computer  monitor  and  operating  a  computer 
keyboard  and  mouse.  Participants  will  participate  in  one  seven  hour  experiment  with  three 
breaks  and  will  be  reimbursed  appropriately. 
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Protocol:  #L-416A 


Title:  Coalition  passive  sonar  track  sharing  and  association 

Principal  Investigator:  Garfield  R.  Mellema,  DRDC  Atlantic 
Co-Investigator:  Tania  E.  Wentzell,  DRDC  Atlantic 

Background 

Coalition  Data  Exchange 

Information  exchange  among  coalition  partners  has  the  potential  to  provide 
significant  advantages  in  the  execution  of  underwater  warfare.  The  degree  to  which  that 
potential  is  realized,  however,  may  be  dependent  on  the  available  bandwidth  between  the 
platforms  and  their  ability  to  process  the  received  data.  As  sensor  data  is  refined  from  sound 
pressure  to  target  location,  course  and  speed,  the  data  volume  decreases  and  its  information 
content  increases.  One  cost  is  latency.  Another  is  the  lost  opportunity  to  undertake  lower- 
level  multi-sensor  processing,  such  as  cross-correlation  or  triangulation. 

Exchanging  sensor-level  sonar  data  entails  its  own  set  of  costs  and  complexities. 

High  bandwidth  connections  as  well  as  powerful  and  sophisticated  processing  techniques  are 
essential  to  the  successful  utilization  of  multiple  streams  of  sensor-level  data.  In  return  one 
gains  the  capability  to  significantly  reduce  the  time  required  to  achieve  target  localization  and 
identification.  The  payoff  is  not  infinite  however,  as  at  some  level  the  costs  of  increased  data 
sharing  outweigh  the  additional  benefits.  In  order  to  make  a  good  decision  as  to  the  most 
appropriate  level  at  which  to  share  data  between  platforms,  one  needs  a  sense  of  how  these 
costs  and  benefits  trade  off. 

The  minimum  level  at  which  it  is  beneficial  to  share  data  depends  strongly  on  the 
speed  and  sophistication  of  the  data  processing  available  at  the  source,  and  the  input 
requirements  of  the  processing  routines  at  the  recipient.  Although  the  minimum  input  level  of 
an  automated  processing  system  may  have  been  clearly  specified,  in  the  case  of  a  human 
operator  the  minimum  beneficial  input  level  may  be  more  difficult  to  identify,  as  it  may  be 
masked  by  issues  related  more  to  operator  loading  than  to  operator  comprehension. 

Investigating  the  potential  benefits  of  low-level  sonar  data  exchange  requires  the 
examination  of  the  results  of  a  series  of  scenarios  in  which  sonar  data  at  differing  levels  of 
development  are  provided  to  the  ownship4  from  a  geographically  remote  allied  vessel.  The 
value  of  these  results  will  be  assessed  in  terms  of  the  quality  of  development  of  the  local 
operating  picture.  The  experiment  described  in  this  document  is  the  first  in  a  series  of 
experiments  aimed  at  implementing  this  process. 

Passive  Sonar  Track  Association 

A  single  target  vessel  will  typically  produce  acoustic  emissions  at  multiple 
frequencies.  These  signals  may  propagate  along  multiple  paths  to  a  receiver  and  may  be 
intermittent  in  time  due  to  such  external  influences  as  variations  in  source  or  receiver  location 


4  Ownship  is  a  commonly  used  naval  term  to  indicate  ‘our  own  ship’,  that  is,  the  vessel  that  is 
controlled  by  the  subject. 
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or  the  local  environment.  Each  emission  is  eventually  represented  to  the  sonar  operator  as  a 
signal  track  segment  from  a  potentially  unique  source. 

Passive  sonar  association  is  the  process  of  associating  signal  track  segments  across 
time,  frequency  and/or  bearing  into  master  tracks  that  are  common  to  a  single  source  vessel. 
Although  rudimentary  tools  exist  to  assist  the  operator  in  this  task,  it  can  be  very  labour 
intensive  and  relies  heavily  on  the  operator’s  training  and  experience.  The  use  of  an 
automated  passive  sonar  association  system  or  aid  has  the  potential  to  improve  situational 
awareness  in  the  underwater  environment  and  reduce  the  response  time  to  threats  by 
increasing  the  effectiveness  of  the  human  operator. 

If  one  has  knowledge  of  the  local  environment,  the  locations  of  a  source  and  a 
receiver  and  the  source  level,  it  can  be  fairly  straightforward  to  calculate  the  sonar  track 
segments  that  would  result.  The  inverse  is  not  necessarily  true,  as  it  can  be  extremely  difficult 
to  reliably  determine  whether  apparent  relationships  between  multiple  track  segments 
correspond  to  a  common  origin. 

In  order  to  provide  some  insight  into  potential  processes  for  automated  track  segment 
association,  a  better  understanding  of  the  decision  process  by  which  a  human  operator  decides 
whether  or  not  to  associate  track  segments  is  required.  The  experiment  described  in  this 
document  is  the  first  in  a  series  of  experiments  aimed  at  investigating  that  process. 

Purpose 

Data  exchange  among  coalition  partners  can  take  place  at  multiple  levels  of 
refinement.  While  highly  refined  data  may  require  little  additional  processing  by  the  operator, 
low-level  data  may  contain  information  lost  in  the  refinement  process.  Deciding  at  which 
level  to  exchange  data  requires  an  understanding  of  the  value  of  the  differing  levels  of  data  to 
the  recipient.  One  goal  of  the  experimentation  campaign,  of  which  this  is  the  first  experiment, 
is  to  investigate  how  the  value  of  coalition  data  sharing  changes  as  the  exchanged  data 
becomes  increasingly  refined.  This  experiment  will  provide  a  baseline  for  comparison,  by 
contrasting  local  operating  picture  development  in  the  cases  of  a  single  sonar  operating  alone, 
and  with  shared  bearings-only  sonar  tracks  from  an  allied  vessel. 

Tracks  developed  from  sonar  sensor  data  can  exhibit  discontinuities  due  to  such 
conditions  as  apparently  erratic  movement  of  the  target  or  a  temporary  reduction  of  the  signal 
to  noise  ratio  of  the  target  below  the  system  detection  threshold.  The  track  segments 
preceding  and  following  the  discontinuity  share  a  common  origin  and  therefore  merit 
association  into  a  common  master  track,  but  it  is  difficult  for  an  automated  system  to 
accurately  and  reliably  identify  such  segment  pairs  in  spite  of  their  apparently  similar 
characteristics.  A  second  goal  of  this  experiment  is  to  investigate  the  decision-making 
process  used  by  the  operator  to  decide  which  track  segments  to  associate  or  disassociate  and 
when  to  do  so.  This  information  will  provide  insight  into  potential  algorithms  for  automated 
track  association  and  a  benchmark  for  evaluating  the  effectiveness  of  automated  track 
association  techniques. 
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Prior  to  the  main  experiment,  a  pilot  experiment  will  be  executed,  the  objective  of 
which  is  to  ensure  the  suitability  and  effectiveness  of  the  process  and  procedures  used  in  the 
main  experiment. 

Selection  of  Subjects 

The  pilot  experiment  will  be  held  at  DRDC  Atlantic.  Participants  will  be  DRDC 
Atlantic  employees,  preferably  Navy  personnel  familiar  with  naval  sonar.  They  will  be 
recruited  using  the  poster  in  Appendix  A  distributed  by  email  or  posted  at  DRDC  Atlantic. 
Twelve  participants  will  be  used  in  order  to  establish  the  variance  of  the  measurement 
process. 


The  main  experiment  will  also  be  held  at  DRDC  Atlantic.  Participants  will  be  Navy 
personnel  familiar  with  naval  sonar,  recruited  from  the  Canadian  Forces  Naval  Operations 
School  (CFNOS)  in  Halifax.  They  will  be  recruited  using  a  poster  distributed  by  email  or 
posted  at  CFNOS.  The  number  of  participants,  which  will  be  determined  by  the  variance  of 
the  results  of  the  pilot  experiment,  is  anticipated  to  be  between  12  and  20. 

Participants  in  both  the  pilot  and  main  experiment  will  be  limited  to  those  between  18 
and  65  years  of  age  and  self-selected  to  have  normal  or  corrected  to  normal  vision.  Although 
this  study  is  gender  neutral,  the  subject  gender  ratio  is  anticipated  to  reflect  the  demographics 
of  the  group  from  which  the  subjects  are  recruited. 

Methodology 

Apparatus 

The  study  will  be  conducted  in  an  office  under  normal  lighting  conditions.  The 
subject  will  be  seated  in  an  office  chair  in  front  of  a  computer  table.  In  the  first  configuration, 
a  keyboard,  mouse  and  a  17”  flat-panel  display  will  allow  the  operator  to  interact  with  a 
computer  running  an  interactive  simulation  of  a  maritime  environment.  In  a  second 
configuration,  an  additional  17”  flat  panel  display  is  added,  along  with  a  second  keyboard  and 
mouse.  These  will  allow  the  operator  to  interact  with  a  second  computer  that  is  running 
additional  components  of  the  maritime  simulation. 

Stimuli 

The  operator  is  presented  with  a  chart  display  showing  the  navigable  boundaries  of  a 
narrow  strait  and  indicating  the  position  of  his  own  ship  as  well  as  that  of  an  allied  vessel  as 
shown  in  Figure  1 1 .  The  ownship  is  following  a  predetermined  course  at  predetermined 
speeds  appropriate  for  the  operator  tasking.  No  other  information  about  other  vessel  traffic  in 
the  strait  is  available  from  the  chart  display.  Passive  sonar  is  being  used  to  monitor  all  vessel 
traffic  in  the  strait,  and  the  sonar  tracks  are  available  as  plots  of  bearing  versus  time  as  shown 
in  Figure  12.  The  sonar  tracks  corresponding  to  the  target  vessels  are  disjoint  in  time,  due  to 
variations  in  the  environment  and  the  acoustic  source  level  of  each  target  as  well  as 
interference  from  other  acoustic  sources.  The  relationship  between  each  target  and  its  one  or 
more  representative  track  segments  is  indistinct,  due  primarily  to  the  lack  of  immediate  range 
information  from  the  passive  sonar  system.  In  the  second  configuration,  passive  sonar  track 
information  in  a  similar  format  is  available  from  a  sensor  aboard  the  allied  vessel  and  shown 
on  the  leftmost  screen. 
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Figure  11  Ownship  chart  display 


Task 

The  task  of  the  operator  is  to  develop  an  operational  picture  of  vessel  traffic  in  the 
strait  using  only  knowledge  of  the  strait  and  of  own  and  allied  ship  positions  from  the  chart 
display  and  passive  sonar  track  information  from  the  track  displays.  In  order  to  refine  the 
ownship  track  display,  the  operator  can  associate  multiple  track  segments  that  are  believed  to 
have  originated  from  the  same  target  vessel  into  a  master  track.  Similarly,  the  operator  may 
disassociate  a  master  track  back  into  its  original  track  segments  should  it  be  concluded  that 
their  association  was  without  merit.  An  ideally  developed  track  display  would  present  a 
single  continuous  track  for  each  target  vessel  for  the  entire  time  that  it  is  within  detection 
range  of  the  ownship  sonar.  In  the  second  configuration,  when  passive  sonar  track 
information  is  available  from  the  allied  vessel,  the  additional,  nonorganic  track  segments  may 
be  associated  amongst  themselves  and  the  results  observed  by  the  operator,  but  cannot  be 
electronically  communicated  onto  the  ownship  display  or  associated  with  the  organic  track 
data.  The  level  of  development  of  the  ownship  track  display  will  determine  operational 
picture  quality. 
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Figure  12  Ownship  passive  sonar  track  display 

With  few  exceptions,  all  operator  interaction  with  the  simulation  can  be  performed 
using  only  the  mouse.  Changes  to  the  scale  of  the  chart  display  can  be  done  using  the  mouse, 
as  can  changes  to  the  time  or  bearing  scale  of  the  track  display.  Track  segments  on  a  track 
display  can  be  associated  into  a  fused  master  track  by  using  the  mouse  to  select  the 
component  segments  and  then  clicking  a  checkmark  icon  within  the  track  association  module. 
Clicking  on  a  corresponding  “X”  icon  will  disassociate  a  selected  master  track. 

Following  either  an  association  or  a  disassociation  action,  the  operator  will  be  queried 
by  a  popup  menu  as  to  the  cognitive  process  that  led  to  the  association  or  disassociation 
decision.  The  operator  will  be  asked  to  indicate  the  rationale  that  led  to  the  decision  from  a 
prepared  list  or  provide  an  alternate  rationale.  A  sample  of  the  initial  list  of  prepared  rationale 
to  be  used  in  the  pilot  experiment  is  shown  in  Table  7.  The  list  to  be  used  in  the  main 
experiment  will  be  developed  from  the  results  of  the  pilot  experiment.  The  subject  will  be 
supervised  at  all  times. 
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1.  Bearing  continuity _ 

2.  Signal  strength _ 

3.  Operational  constraints  of  the  target  vessel _ 

4.  Operational  constraints  of  the  water  space _ 

5.  Navigational  procedures _ 

6.  Other  (specify) _ 

Table  7  Initial  list  of  prepared  rationales  for  track  association  or  disassociation. 


Procedure 

All  of  the  scenarios  in  this  experiment  take  place  in  the  same  strait,  but  may  differ  in 
the  number,  course  and  speeds  of  the  target  vessels;  the  course  and  speeds  of  the  own  and 
allied  ships;  and  the  ability  to  share  sonar  data  with  the  allied  ship. 

The  experiment  will  begin  with  a  training  session  in  which  the  operator  is  introduced 
to  the  simulation  system  and  the  two  types  of  scenarios.  During  this  training  session,  the 
operator  will  be  introduced  to  the  navigational  chart  of  the  strait,  and  to  the  ownship  and 
allied  ship  sonar  track  displays.  The  controls  to  modify  the  resolution  of  the  screen  displays 
will  be  demonstrated,  as  will  the  controls  to  associate  and  disassociate  track  segments.  The 
operator  will  then  have  the  opportunity  to  interact  with  a  version  of  the  second  experimental 
configuration,  i.e.  including  shared  data  from  the  allied  ship,  which  differs  from  the  testing 
versions.  The  total  time  allocated  for  subject  training  is  one  hour.  Prior  to  the  training,  the 
subject  will  be  asked  to  sign  the  consent  form  in  Appendix  B. 

Following  operator  training  and  familiarization,  the  subject  will  participate  in  two 
testing  sessions  employing  the  methodology  described  above.  Each  subject  will  interact  with 
both  the  independent  and  coalition  simulations,  in  random  order.  Each  simulation  will  last 
about  two  hours. 

A  closing  interview  will  follow  the  simulations.  During  the  interview,  the  results  of 
the  test  may  be  discussed  with  the  subject.  They  will  be  asked  to  describe  the  cues  they 
thought  to  be  useful  in  deciding  what  to  associate  or  disassociate  and  when  to  do  so.  These 
cues  may  be  added  to  later  versions  of  the  (dis)association  rationale  checklist.  The  subject 
will  also  be  asked  to  describe  his  perception  of  the  value  of  the  sonar  tracks  shared  by  the 
allied  vessel.  The  interview  is  estimated  to  last  about  one  half  hour. 

Analysis 

Quantitative  Analysis 

The  level  of  development  of  the  local  operating  picture  will  be  evaluated  based  on  the 
number  and  configuration  of  tracks  on  the  ownship  track  display.  Four  measures  will  be  used. 

1.  Clarity  measures  the  number  of  displayed  tracks  relative  to  the  number  of  track 
segments  available  to  be  displayed.  Recall  that  association  replaces  multiple 
track  segments  from  the  display  and  replaces  them  with  a  single  master  track. 

2.  Completeness  measures  the  number  of  correct  associations  made  versus  the  total 
number  of  possible  correct  associations. 
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3.  Continuity  measures  the  number  of  disjoint  track  segments  displayed  relative  to 
the  number  of  target  vessels.  Note  that  in  this  experiment,  each  vessel  can 
produce  no  more  than  one  sonar  track  at  a  time. 

4.  Correctness  measures  the  number  of  correct  associations  made  relative  to  the  to 
the  total  number  of  associations  made. 

The  relative  levels  of  development  of  the  local  operating  picture  will  be  compared  with  and 
without  sonar  track  data  shared  by  an  allied  vessel  to  determine  the  value  of  coalition  data 
sharing  at  this  level.  The  variance  in  this  measure  of  improvement  will  be  used  to  determine 
the  number  of  subjects  required  for  the  main  experiment. 

Qualitative  Measures 

The  number  of  times  that  each  of  the  rationales  was  cited  by  an  operator  as  a  factor  in 
his  decision  to  associate  two  or  more  track  segments  into  a  master  track  will  be  analyzed  to 
determine  its  relative  value  to  the  decision  making  process.  The  prepared  list  of  association 
rationales  offered  to  the  operator  will  be  revised  based  on  the  number  of  times  each  rationale 
was  cited  in  the  pilot  experiment.  Those  rationales  that  were  heavily  cited  may  be  broken 
down  into  subcategories.  Rationales  may  be  added  or  removed  from  the  list  used  in  the  main 
experiment  based  on  indications  under  the  “other”  category  and  feedback  from  the  closing 
interviews  in  the  pilot  experiment. 

A  record  of  the  execution  of  each  experimental  scenario  will  be  made  which  will  be 
suitable  for  reproducing  the  displays  seen  and  the  actions  taken  by  the  operator.  Comments 
made  by  the  subjects  during  the  closing  interview  also  will  be  recorded. 

Medical  Screening  and  Physician  Coverage 

No  medical  screening  or  physician  coverage  is  required  because  this  is  a  minimal  risk 

study. 

Risks 


This  experiment  poses  only  minimal  risk  to  the  participants.  No  psychological 
measures  will  be  taken  and  no  emotionally  distressing  stimuli  will  be  used.  Participants  are 
required  to  remain  attentive.  There  is  a  risk  of  eyestrain  and  fatigue  associated  with  viewing  a 
computer  monitor  and  operating  a  computer  keyboard  and  mouse,  but  no  greater  than  those 
associated  with  everyday  computer  use.  Subjects  will  be  free  to  leave  at  any  time.  Data  files 
and  statements  of  performance  will  at  no  time  be  associated  with  the  subject's  identification 
to  maintain  anonymity. 

Approximate  Time  Involvement 

Subjects  will  be  asked  to  participate  in  a  training  session  lasting  about  one  hour,  two 
simulations  each  lasting  about  two  hours,  and  an  interview  lasting  about  half  an  hour.  All  of 
these  activities  can  be  scheduled  into  a  single  day  including  two  15  minute  breaks  and  a  half 
hour  minute  lunch  break.  The  total  time  involvement  will  not  exceed  seven  hours. 

Remuneration 
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Participants  will  be  reimbursed  for  their  participation  according  to  the  DRDC  Toronto 
stress  allowance:  $2.50  (Level  I  stress)  x  7  hours  +  $10.14  (daily  rate)  =  $27.64. 

Benefits 

This  experiment  will  provide  an  opportunity  to  evaluate  the  feasibility  and 
effectiveness  of  the  described  experimental  technique  in  the  pursuit  of  the  indicated 
experimental  objectives. 

Comparison  of  the  relative  levels  of  operational  picture  development  between  the 
ownship  alone,  and  ownship  and  cooperative  allied  ship  scenarios  will  provide  insight  into  the 
value  of  coalition  sonar  track  data  sharing.  This  experiment  is  the  first  of  a  planned  series  of 
experiments  to  investigate  the  value  of  sonar  data  sharing  among  coalition  partners.  In  later 
experiments,  these  results  will  be  used  as  a  baseline  for  the  evaluation  of  other  data 
integration  techniques  at  this  and  other  levels  of  data  development. 

The  cues  and  rationales  employed  by  sonar  operator  in  the  process  of  passive  sonar 
track  association  and  disassociation  can  provide  some  direction  in  the  development  of 
algorithms  for  automated  passive  sonar  track  association.  The  availability  of  an  automated 
passive  sonar  track  association  system  could  significantly  reduce  operator  loading  and 
decrease  operator  response  time,  especially  in  target-rich  environments. 

This  experiment  offers  the  participant  an  opportunity  to  share  his  or  her  expertise  and 
participate  in  the  development  of  future  sonar  analysis.  The  subjects  will  also  be  reimbursed 
for  their  participation. 

Commercialization 

The  research  findings  resulting  from  this  work  could  be  used  for  commercialization 
purposes. 

References 

Mellema,  G.R.,  “A  methodology  for  the  investigation  and  development  of  automated 
passive  sonar  track  association,”  Defence  Research  and  Development  Canada  -  Atlantic, 
Canada,  Technical  Memorandum  DRDC  Atlantic  TM  2002-195,  Dec  2002. 

Pigeau,  R.  (1992).  Command  Group’s  Guide  to  stress  compensation  for  human 
subjects.  Unpublished  DRDC  Document,  DRDC  Toronto. 

“DRDC  Toronto  guidelines  for  human  subject  participation  in  research  projects,” 
Defence  Research  and  Development  Canada  -  Toronto,  Apr  2002. 


66 


DRDC  Atlantic  TM  2003-158 


APPENDIX  A 


RECRUITMENT  POSTER 


RESEARCH  PARTICIPANTS  REQUIRED 

Coalition  passive  sonar  track  sharing  and  association 

PARTICIPANTS 

Males  and  females  between  18  and  65  years  of  age  with  normal  or  corrected 

to  normal  vision. 

OBJECTIVES 

This  experiment  will  study  the  value  of  low-level  coalition  data  sharing  in  the  development  of 

local  operational  picture. 

PROCEDURE 

Simulated  sonar  tracks  similar  to  those  presented  by  a  passive  towed  array  sonar  system  will 
be  presented  with  or  without  corresponding  tracks  from  an  allied  vessel  at  a  geographically 
remote  location.  The  participant’s  task  is  to  use  the  available  tracks  as  well  as  a  navigational 
chart  display  and  an  initial  situation  report  to  develop  a  local  operating  picture. 

The  session  will  last  approximately  7  hours  and  includes  some  basic  training. 

WHEN 

During  normal  working  hours  in  September  and  October  2003. 

WHERE 

DRDC  Atlantic,  Number  9  Grove  St.,  Dartmouth,  Nova  Scotia 

COMPENSATION 

Participants  will  be  compensated  according  to  DRDC  guidelines. 

For  more  information,  all  interested  volunteers  should  contact: 

Garfield  Mellema  at  (902)  426-3100  x  252  /  garfield.mellema@drdc-rddc.gc.ca 
Tania  Wentzell  at  (902)  426-3100  x283  /  tania.wentzell@drdc-rddc.gc.ca 
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APPENDIX  B 

VOLUNTARY  CONSENT  FORM  FOR  HUMAN  SUBJECT  PARTICIPATION 
Protocol  Number:  L-416 

Research  Project  Title:  Coalition  passive  sonar  track  sharing  and  association 
Principal  Investigator:  Garfield  R.  Mellema 
Co-investigator(s):  Tania  E.  Wentzell 

I, _ (name)  of _ (address 

and  phone  number)  hereby  volunteer  to  participate  as  a  subject  in  the  study,  “Coalition 
passive  sonar  track  sharing  and  association”  (Protocol  #  L-416).  I  have  read  the  information 
package  on  the  research  protocol,  and  have  had  the  opportunity  to  ask  questions  of  the 
Investigators.  All  of  my  questions  concerning  this  study  have  been  fully  answered  to  my 
satisfaction.  However,  I  may  obtain  additional  information  about  the  research  project  and 
have  any  questions  about  this  study  answered  by  contacting  the  principle  investigator  Dr. 
Garfield  R.  Mellema  at  (902)  426-3100  ext  252. 

I  have  been  told  that  I  will  be  asked  to  participate  in  one  session  of  approximately  seven  hours 
in  duration. 

I  have  been  told  that  the  principal  risks  of  the  research  protocol  are:  possible  minor  eyestrain 
and  fatigue. 

Also,  I  acknowledge  that  my  participation  in  this  study,  or  indeed  any  research,  may  involve 
risks  that  are  currently  unforeseen  by  DRDC  Atlantic. 

For  Canadian  Forces  (CF)  members  only:  I  understand  that  I  am  considered  to  be  on  duty  for 
disciplinary,  administrative  and  Pension  Act  purposes  during  my  participation  in  this 
experiment  and  I  understand  that  in  the  unlikely  event  that  my  participation  in  this  study 
results  in  a  medical  condition  rendering  me  unfit  for  service,  I  may  be  released  from  the  CF 
and  my  military  benefits  apply.  This  duty  status  has  no  effect  on  my  right  to  withdraw  from 
the  experiment  at  any  time  I  wish  and  I  understand  that  no  action  will  be  taken  against  me  for 
exercising  this  right. 

I  have  been  advised  that  the  experimental  data  concerning  me  will  be  treated  as  confidential 
(‘Protected  B’  IAW  CF  Security  Requirements),  and  not  revealed  to  anyone  other  than  the 
DRDC  Atlantic  Investigator(s)  or  external  investigators  from  the  sponsoring  agency  without 
my  consent  except  as  data  unidentified  as  to  source.  Moreover,  should  it  be  required,  I  agree 
to  allow  the  experimental  data  to  be  reviewed  by  an  internal  or  external  audit  committee  with 
the  understanding  that  any  summary  information  resulting  from  such  a  review  will  not 
identify  me  personally. 

I  understand  that  I  am  free  to  refuse  to  participate  and  may  withdraw  my  consent  without 
prejudice  or  hard  feelings  at  any  time.  Should  I  withdraw  my  consent,  my  participation  as  a 
subject  will  cease  immediately,  unless  the  Investigator(s)  determine  that  such  action  would  be 
dangerous  or  impossible  (in  which  case  my  participation  will  cease  as  soon  as  it  is  safe  to  do 
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so).  I  also  understand  that  the  Investigator(s)  or  their  designate  responsible  for  the  research 
project  may  terminate  my  participation  at  any  time,  regardless  of  my  wishes. 

I  have  been  informed  that  the  research  findings  resulting  from  my  participation  in  this 
research  project  could  be  used  for  commercialization  purposes. 

I  understand  that  for  my  participation  in  this  research  project,  I  am  entitled  to  remuneration  in 
the  form  of  a  stress  allowance  in  the  amount  of  $27.64  for  each  completed  session  for  a  total 
amount  of  $27.64  if  I  complete  the  entire  research  project. 

I  have  informed  the  Principal  Investigator  that  I  am  currently  a  subject  in  the  following  other 

DRDC  Atlantic  research  project(s): _ 

(cite  Protocol  Number(s)  and  associated  Principal  Investigator(s)),  and  that  I  am  participating 
as  a  subject  in  the  following  research  project(s)  at  institutions  other  than  DRDC  Atlantic: 
_ (cite  name(s)  of  institution(s)) 

I  understand  that  by  signing  this  consent  form  I  have  not  waived  any  legal  rights  I  may  have 
as  a  result  of  any  harm  to  me  occasioned  by  my  participation  in  this  research  project  beyond 
all  risks  I  have  assumed. 

Volunteer’s  Name  _  Signature:  _ 


Date: _ 

Name  of  Witness  to  Signature: _ 

Signature: _ Date: _ 

Section  Head/Commanding  Officer’s  Signature  (see  Notes  below) _ 

CO’s  Unit: _ 

Principal  Investigator:  Dr.  Garfield  R.  Mellema,  DRDC  Atlantic 
Signature: _ 

Date: _ 

FOR  SUBJECT  ENQUIRY  IF  REQUIRED: 

Should  I  have  any  questions  or  concern  regarding  this  project  before,  during,  or  after 
participation,  I  understand  that  I  am  encouraged  to  contact  Defence  R&D  Canada  (DRDC). 
This  contact  can  be  made  by  surface  mail  at  this  address  or  in  person,  by  phone  or  e-mail,  to 
any  of  the  DRDC  numbers  and  addresses  listed  below: 

Principle  Investigator  (DRDC  Atlantic): 

Dr.  Garfield  R.  Mellema,  (902)  426-3100,  garfield.mellema@drdc-rddc.gc.ca 
Mail  Address: 
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Dr.  Garfield  R.  Mellema,  Defence  R&D  Canada  -  Atlantic,  PO  Box  1012,  9  Grove  St., 
Dartmouth,  Nova  Scotia,  Canada  B2Y  3Z7 

Chair,  DRDC  Toronto  Human  Research  Ethics  Committee  (HREC): 

Dr.  Jack  Landolt,  (416)  635-2120,  jack.landolt@drdc-rddc.gc.ca 
Mail  Address: 

Dr.  Jack  Landolt,  Defence  R&D  Canada  -  Toronto,  PO  Box  2000,  1133  Sheppard  Avenue 
West,  Toronto,  Ontario  M3M  3B9 

I  understand  that  I  will  be  given  a  copy  of  this  consent  form  so  that  I  may  contact  any  of  the 
above-mentioned  individuals  at  some  time  in  the  future  should  that  be  required. 


Notes: 

For  Military  personnel  on  permanent  strength  of  CFEME:  Approval  in  principle  by 
Commanding  Officer  is  given  in  Memorandum  3700-1  (CO  CFEME),  18  Aug  94;  however, 
members  must  still  obtain  their  Section  Head’s  signature  designating  approval  to  participate  in 
this  particular  research  project. 

For  other  military  personnel:  All  other  military  personnel  must  obtain  their  Commanding 
Officer’s  signature  designating  approval  to  participate  in  this  research  project. 

For  civilian  personnel  at  DRDC  Toronto:  Signature  of  Section  Head  is  required  designating 
that  the  volunteer  subject  is  considered  to  be  at  work  and  that  approval  has  been  given  to 
participate  in  this  research  project. 
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List  of  acronyms 


C2 

Command  and  Control 

DSTO 

(The  Australian)  Defence  Science  and  Technology  Organization 

HLA 

High  Level  Architecture 

HREC 

Human  Research  Ethics  Committee  (at  DRDC  Toronto) 

TM 

Information  Management 

MAR 

Maritime  Systems  Group  (of  TTCP) 

RTI 

Run-Time  Interface 

TMA 

Target  Motion  Analysis 

TP-1 

Technical  Panel  1  (of  TTCP  MAR) 

TTCP 

The  Technical  Cooperation  Panel 

UDF 

Underwater  Warfare  Data  Fusion  (Group) 

VBE 

Virtual  Battle  Experiment 

VCS 

Virtual  Combat  Systems  (Group) 

VMSEM 

Virtual  Maritime  System  Execution  Manager 

VMSA 

Virtual  Maritime  Systems  Architecture 

VMSSD 

Virtual  Maritime  System  Simulation  Display 
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1 3.  ABSTRACT  (a  brief  and  factual  summary  of  the  document.  It  may  also  appear  elsewhere  in  the  body  of  the  document  itself.  It  is 
highly  desirable  that  the  abstract  of  classified  documents  be  unclassified.  Each  paragraph  of  the  abstract  shall  begin  with  an  indication 
of  the  security  classification  of  the  information  in  the  paragraph  (unless  the  document  itself  is  unclassified)  represented  as  (S),  (C),  (R), 
or  (U).  It  is  not  necessary  to  include  here  abstracts  in  both  official  languages  unless  the  text  is  bilingual). 

This  document  outlines  the  experimental  plan  for  the  first  Canadian  Virtual  Battle  Experiment, 
VBE  CA-1.  It  is  intended  to  provide  some  context  for  the  experiment  and  act  as  a  blueprint 
and  guide  for  its  implementation  and  execution. 

VBEs  are  being  used  by  the  Maritime  Systems  Group  Technical  Panel  1  of  The  Technical 
Cooperation  Panel  (TTCP  MAR  TP-1)  to  investigate  the  influence  of  Network  Enabled 
Capability  in  a  modular  synthetic  maritime  environment.  VBE  CA-1  is  the  first  of  a  series  of 
experiments  investigating  the  effect  of  sharing  low-level  passive  sonar  data.  It  was  designed 
to  use  multiple  subjects  in  multiple  sessions  of  a  human-in-the-loop  experiment  to  produce 
statistically  relevant  results. 
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If  possible  keywords  should  be  selected  from  a  published  thesaurus,  e.g.  Thesaurus  of  Engineering  and  Scientific  Terms  (TEST)  and 
that  thesaurus-identified.  If  it  not  possible  to  select  indexing  terms  which  are  Unclassified,  the  classification  of  each  should  be 
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